Oct 30 06:53:58.546268 ip-10-0-143-90 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 06:53:58.546282 ip-10-0-143-90 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 06:53:58.546291 ip-10-0-143-90 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 06:53:58.546603 ip-10-0-143-90 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 06:54:08.725181 ip-10-0-143-90 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 06:54:08.725197 ip-10-0-143-90 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d215f6d4631e44318bb592039bf0d711 -- Oct 30 06:56:02.552928 ip-10-0-143-90 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:56:03.067315 ip-10-0-143-90 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:03.067315 ip-10-0-143-90 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:56:03.067315 ip-10-0-143-90 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:03.067315 ip-10-0-143-90 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:56:03.076435 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076337 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.076435 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076431 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.076435 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076437 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.076435 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076441 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.076435 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076444 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076447 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076451 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076453 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076456 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076459 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076461 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076465 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076468 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076470 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076473 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076475 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076478 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076481 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076483 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076486 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076489 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076492 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076495 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076498 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.076663 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076500 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076514 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076517 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076520 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076522 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076525 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076528 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076530 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076533 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076536 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076540 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076544 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076548 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076551 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076554 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076557 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076560 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076563 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076565 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076568 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.077130 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076571 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076573 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076576 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076578 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076581 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076584 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076587 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076590 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076601 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076605 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076607 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076610 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076613 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076615 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076618 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076620 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076623 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076626 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076629 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076632 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.077607 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076650 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076654 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076658 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076662 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076667 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076670 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076673 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076676 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076679 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076682 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076684 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076687 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076689 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076692 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076695 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076697 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076700 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076704 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076708 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.078093 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076713 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076716 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076718 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076721 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076724 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076727 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076729 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076732 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076735 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076737 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076741 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076743 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076746 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076749 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076751 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076754 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076757 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076760 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076764 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.078622 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076766 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076769 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076772 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.076774 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077578 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077586 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077589 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077592 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077595 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077597 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077600 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077602 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077605 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077608 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077611 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077614 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077617 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077619 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077622 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.079080 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077625 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077627 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077630 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077647 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077651 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077654 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077657 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077659 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077662 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077665 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077667 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077671 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077674 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077677 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077680 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077683 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077685 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077688 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077690 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.079544 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077693 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077696 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077699 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077701 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077704 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077707 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077709 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077712 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077714 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077717 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077720 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077722 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077725 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077728 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077730 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077732 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077735 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077738 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077741 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077744 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.080002 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077746 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077748 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077751 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077753 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077757 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077762 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077766 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077769 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077772 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077775 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077778 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077780 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077783 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077786 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077789 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077791 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077794 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077796 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077799 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077802 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.080483 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077804 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077807 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077810 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077813 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077816 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077818 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077821 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077824 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077827 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077829 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077832 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077834 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077837 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077839 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077842 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077844 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077848 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077851 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077853 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.080986 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077856 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077859 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077862 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077864 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077867 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077871 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077876 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077879 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077882 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077886 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077889 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077893 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.077896 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078387 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078392 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078396 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078399 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078402 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078404 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.081471 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078407 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078410 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078412 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078415 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078418 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078420 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078423 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078425 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078429 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078432 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078436 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078439 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078442 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078445 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078448 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078450 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078453 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078455 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078458 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078460 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.081948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078464 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078466 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078469 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078471 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078474 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078476 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078479 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078481 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078494 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078498 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078501 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078503 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078506 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078508 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078511 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078515 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078519 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078522 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078525 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.082407 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078528 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078531 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078534 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078537 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078540 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078543 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078545 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078548 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078550 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078553 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078556 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078558 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078561 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078563 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078566 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078568 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078571 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078574 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078576 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078578 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.082868 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078581 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078584 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078586 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078590 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078592 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078595 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078598 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078601 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078603 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078606 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078608 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078611 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078614 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078617 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078619 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078622 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078624 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078627 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078630 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078646 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.083339 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078649 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078652 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078655 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078658 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078660 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078663 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078665 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078668 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078671 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078673 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078676 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078678 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078681 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078684 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078687 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078690 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078693 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078696 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078698 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.083824 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078701 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.078703 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079808 2566 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079819 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079827 2566 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079833 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079842 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079845 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079850 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079854 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079858 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079861 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079865 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079868 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079872 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079875 2566 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079878 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079882 2566 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079885 2566 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079888 2566 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079892 2566 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079895 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079898 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079901 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:56:03.084267 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079905 2566 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079909 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079912 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079916 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079919 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079922 2566 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079929 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079932 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079935 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079938 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079943 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079946 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079949 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079952 2566 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079955 2566 flags.go:64] FLAG: --enable-server="true" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079958 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079963 2566 flags.go:64] FLAG: --event-burst="100" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079967 2566 flags.go:64] FLAG: --event-qps="50" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079970 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079973 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079976 2566 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079981 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079984 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079987 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079990 2566 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:56:03.084848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079993 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079996 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.079999 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080002 2566 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080006 2566 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080009 2566 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080012 2566 flags.go:64] FLAG: --feature-gates="" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080017 2566 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080021 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080025 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080028 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080031 2566 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080035 2566 flags.go:64] FLAG: --help="false" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080039 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080043 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080046 2566 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080049 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080053 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080056 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080059 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080062 2566 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080065 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080068 2566 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:56:03.085457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080071 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080074 2566 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080077 2566 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080080 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080083 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080086 2566 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080090 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080093 2566 flags.go:64] FLAG: --lock-file="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080096 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080099 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080102 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080108 2566 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080111 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080114 2566 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080117 2566 flags.go:64] FLAG: --logging-format="text" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080120 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080124 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080129 2566 flags.go:64] FLAG: --manifest-url="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080132 2566 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080137 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080140 2566 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080145 2566 flags.go:64] FLAG: --max-pods="110" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080150 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080153 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080156 2566 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:56:03.086084 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080160 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080163 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080165 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080168 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080176 2566 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080179 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080182 2566 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080185 2566 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080188 2566 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080191 2566 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080194 2566 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080197 2566 flags.go:64] FLAG: --port="10250" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080200 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080203 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b11a7f7464b1f6c4" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080206 2566 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080209 2566 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080212 2566 flags.go:64] FLAG: --register-node="true" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080215 2566 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080219 2566 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080222 2566 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080225 2566 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080228 2566 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080232 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080235 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080239 2566 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080242 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:56:03.086680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080245 2566 flags.go:64] FLAG: --runonce="false" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080249 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080252 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080257 2566 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080260 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080263 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080267 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080270 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080273 2566 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080276 2566 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080279 2566 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080282 2566 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080285 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080288 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080291 2566 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080294 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080299 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080302 2566 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080305 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080309 2566 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080312 2566 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080315 2566 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080318 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080321 2566 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080324 2566 flags.go:64] FLAG: --v="2" Oct 30 06:56:03.087283 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080328 2566 flags.go:64] FLAG: --version="false" Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080333 2566 flags.go:64] FLAG: --vmodule="" Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080337 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080341 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080477 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080482 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080485 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080487 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080490 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080493 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080496 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080499 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080502 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080505 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080508 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080510 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080513 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080516 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080518 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080521 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.087894 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080523 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080526 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080528 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080531 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080533 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080535 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080538 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080541 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080543 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080546 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080548 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080551 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080553 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080556 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080560 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080563 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080566 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080569 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080572 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.088368 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080574 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080578 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080582 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080585 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080588 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080591 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080602 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080605 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080609 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080612 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080614 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080617 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080620 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080622 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080625 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080627 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080630 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080645 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080650 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.088858 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080654 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080658 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080662 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080667 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080670 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080673 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080676 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080678 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080681 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080684 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080686 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080688 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080691 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080693 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080696 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080698 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080701 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080704 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080706 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080709 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.089317 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080711 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080714 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080717 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080720 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080723 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080725 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080728 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080730 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080733 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080735 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080738 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080740 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080743 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080745 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080748 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080750 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080753 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080755 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080758 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.089809 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080760 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080763 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080765 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080767 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080770 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080772 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080775 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080777 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080780 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080783 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080785 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080788 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080790 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.080819 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080863 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080869 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.090309 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080872 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080875 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080877 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080880 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080883 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080886 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080889 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080893 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080897 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080900 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080903 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080905 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080909 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080913 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080916 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080919 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080922 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080924 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080927 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.090704 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080930 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080933 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080935 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080938 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080940 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080943 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080946 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080950 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080953 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080955 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080958 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080961 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080963 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080966 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080971 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080973 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080976 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080978 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080981 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.091150 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080984 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080986 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080989 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080992 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080994 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.080997 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081001 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081004 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081007 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081009 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081012 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081014 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081017 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081019 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081022 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081024 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081027 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081029 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081032 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081034 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.091599 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081038 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081041 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081043 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081046 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081048 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081051 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081054 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081056 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081060 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081063 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081065 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081068 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081070 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081073 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081076 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081078 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081081 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081083 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081088 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.092140 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081090 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081093 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081096 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081098 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081101 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081104 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081106 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081109 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081111 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081114 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081116 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081119 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081121 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081126 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081129 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081131 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081134 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081136 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081139 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081141 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.092580 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081144 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081146 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081149 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081152 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081154 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081157 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.081159 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.087994 2566 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.088013 2566 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.088609 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.088616 2566 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088692 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088698 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088701 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088704 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088707 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088710 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088712 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088715 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.093064 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088718 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088720 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088723 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088726 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088729 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088731 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088735 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088738 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088740 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088743 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088746 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088749 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088751 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088754 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088757 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088759 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088762 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088764 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088768 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088770 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.093506 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088773 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088777 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088779 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088782 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088784 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088787 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088789 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088792 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088795 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088797 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088800 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088803 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088806 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088808 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088811 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088814 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088816 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088819 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088822 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088824 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.093976 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088827 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088830 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088832 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088835 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088837 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088840 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088842 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088845 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088848 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088850 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088853 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088856 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088858 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088861 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088865 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088867 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088870 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088873 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088875 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088878 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.094431 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088880 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088883 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088886 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088888 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088891 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088894 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088896 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088899 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088903 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088908 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088911 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088914 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088917 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088919 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088922 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088925 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088928 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088930 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088933 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.094963 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088935 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088938 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088940 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088943 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088946 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088949 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088953 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088956 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088959 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088962 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088965 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088968 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088970 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088972 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088975 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088978 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088980 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088983 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.095450 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.088986 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.089014 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089167 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089174 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089177 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089181 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089183 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089186 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089189 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089191 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089194 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089196 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089199 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089201 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089204 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089207 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.095881 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089209 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089212 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089215 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089217 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089220 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089222 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089225 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089228 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089231 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089234 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089236 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089238 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089241 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089243 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089246 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089249 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089251 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089253 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089257 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089260 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.096257 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089262 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089265 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089267 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089270 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089272 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089275 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089277 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089280 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089282 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089285 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089287 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089290 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089293 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089295 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089298 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089300 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089303 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089305 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089308 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.096736 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089310 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089313 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089316 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089319 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089321 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089324 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089327 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089329 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089332 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089335 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089337 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089340 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089342 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089345 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089348 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089350 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089353 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089355 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089358 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089360 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.097216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089363 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089366 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089369 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089371 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089374 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089376 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089379 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089381 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089384 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089386 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089389 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089391 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089394 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089397 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089400 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089402 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089405 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089407 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089410 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.097701 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089412 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089415 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089417 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089420 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089423 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089425 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089427 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089430 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089433 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089435 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089438 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089440 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089442 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089446 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.089471 2566 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089503 2566 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:56:03.098148 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089507 2566 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089511 2566 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089514 2566 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089517 2566 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089519 2566 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089522 2566 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089524 2566 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089527 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089529 2566 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089532 2566 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089535 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089537 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089540 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089543 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089545 2566 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089548 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089551 2566 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089553 2566 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089556 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:56:03.098523 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089559 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089561 2566 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089564 2566 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089566 2566 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089569 2566 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089571 2566 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089573 2566 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089576 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089578 2566 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089581 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089583 2566 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089586 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089588 2566 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089590 2566 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089593 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089596 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089598 2566 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089601 2566 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089603 2566 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089605 2566 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:56:03.099117 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089608 2566 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089611 2566 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089613 2566 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089616 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089618 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089621 2566 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089624 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089626 2566 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089629 2566 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089632 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089646 2566 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089649 2566 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089651 2566 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089654 2566 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089656 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089659 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089662 2566 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089664 2566 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089666 2566 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:56:03.099579 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089669 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089671 2566 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089674 2566 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089676 2566 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089679 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089682 2566 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089684 2566 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089687 2566 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089690 2566 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089692 2566 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089695 2566 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089697 2566 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089700 2566 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089702 2566 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089706 2566 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089710 2566 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089714 2566 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089718 2566 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089721 2566 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:56:03.100029 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089723 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089726 2566 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089729 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089732 2566 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089735 2566 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089737 2566 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089740 2566 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089742 2566 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089745 2566 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089748 2566 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089750 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089753 2566 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089756 2566 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089758 2566 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089761 2566 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089763 2566 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089766 2566 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089768 2566 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089773 2566 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089776 2566 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:56:03.100477 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089778 2566 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089781 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089783 2566 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089786 2566 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089788 2566 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089791 2566 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089793 2566 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:03.089796 2566 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.089860 2566 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.092005 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.092856 2566 server.go:1014] "Starting client certificate rotation" Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.092957 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.093015 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:56:03.100959 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.095414 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:03.104458 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.104436 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:56:03.119752 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.119736 2566 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:56:03.123383 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.123367 2566 log.go:25] "Validated CRI v1 image API" Oct 30 06:56:03.124241 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.124215 2566 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:56:03.126848 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.126821 2566 fs.go:135] Filesystem UUIDs: map[35df0802-4ff4-46f6-9add-e53b2277e98c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 846e5293-d76a-4dff-8954-09f47d5aeeda:/dev/nvme0n1p4] Oct 30 06:56:03.126932 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.126847 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 06:56:03.133727 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.133567 2566 manager.go:217] Machine: {Timestamp:2026-10-30 06:56:03.131826088 +0000 UTC m=+0.436790888 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101333 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8761dc9b45bed08e3a82ed415a61 SystemUUID:ec2b8761-dc9b-45be-d08e-3a82ed415a61 BootID:d215f6d4-631e-4431-8bb5-92039bf0d711 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:a7:64:5b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:a7:64:5b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7e:61:d8:64:6d:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:56:03.133815 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.133734 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:56:03.133913 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.133890 2566 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:56:03.137108 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137083 2566 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:56:03.137243 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137110 2566 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:56:03.137292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137246 2566 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:56:03.137292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137254 2566 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:56:03.137292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137283 2566 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:03.137364 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.137302 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:56:03.138386 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.138375 2566 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:56:03.142574 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.142561 2566 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:56:03.142702 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.142693 2566 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:56:03.143383 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143375 2566 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:56:03.143418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143388 2566 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:56:03.143418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143398 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:56:03.143418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143408 2566 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:56:03.143418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143417 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:56:03.143780 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.143766 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:03.146876 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.146860 2566 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:56:03.157000 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.156983 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:56:03.158366 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158353 2566 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:56:03.158411 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158384 2566 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:56:03.158551 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158539 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:56:03.158585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158562 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:56:03.158585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158571 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:56:03.158647 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158620 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:56:03.158679 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158652 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:56:03.158679 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.158659 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:56:03.159116 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159108 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:56:03.159149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159118 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:56:03.159149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159126 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:56:03.159149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159131 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:56:03.159149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159145 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:56:03.159320 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.159154 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:56:03.160527 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.160519 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:56:03.160566 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.160530 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:56:03.168522 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.168506 2566 server.go:1263] "Started kubelet" Oct 30 06:56:03.168672 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.168612 2566 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:56:03.172744 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.172668 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:56:03.172852 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.172773 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:56:03.173038 ip-10-0-143-90 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:56:03.174631 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.174512 2566 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:56:03.176318 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.176301 2566 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:56:03.178738 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.178719 2566 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 06:56:03.182418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.182400 2566 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:56:03.182418 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.182415 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:56:03.182572 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.182549 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:03.183149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183003 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:56:03.183149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183043 2566 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:56:03.183149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183059 2566 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:56:03.183344 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183298 2566 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:56:03.183344 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183312 2566 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:56:03.183603 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183586 2566 factory.go:153] Registering CRI-O factory Oct 30 06:56:03.183603 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183602 2566 factory.go:223] Registration of the crio container factory successfully Oct 30 06:56:03.183733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183665 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:56:03.183733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183674 2566 factory.go:55] Registering systemd factory Oct 30 06:56:03.183733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183680 2566 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:56:03.183733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183693 2566 factory.go:103] Registering Raw factory Oct 30 06:56:03.183733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.183703 2566 manager.go:1199] Started watching for new ooms in manager Oct 30 06:56:03.184603 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.184577 2566 manager.go:319] Starting recovery of all containers Oct 30 06:56:03.196127 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.196077 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-90.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 06:56:03.196358 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.196324 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:56:03.196445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.196428 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bq9xr" Oct 30 06:56:03.196702 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.196690 2566 manager.go:324] Recovery completed Oct 30 06:56:03.200567 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.200549 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bq9xr" Oct 30 06:56:03.200825 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.196083 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-90.ec2.internal.18e33c4989f61f70 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-90.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-90.ec2.internal,},FirstTimestamp:2026-10-30 06:56:03.16847704 +0000 UTC m=+0.473441834,LastTimestamp:2026-10-30 06:56:03.16847704 +0000 UTC m=+0.473441834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-90.ec2.internal,}" Oct 30 06:56:03.201021 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.201010 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.203064 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203048 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.203144 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203076 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.203144 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203086 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.203625 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203612 2566 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:56:03.203625 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203623 2566 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:56:03.203742 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203689 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.203742 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.203695 2566 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:56:03.204967 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.204950 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.205035 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.204980 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.205035 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.204990 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.205972 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.205947 2566 policy_none.go:50] "Start" Oct 30 06:56:03.206058 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.205978 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:56:03.206058 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.205992 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:56:03.208235 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.208221 2566 policy_none.go:44] "Start" Oct 30 06:56:03.212982 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.212956 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 06:56:03.212982 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.212971 2566 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-90.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 06:56:03.213109 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.213063 2566 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 06:56:03.243371 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.243351 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:03.255273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.255240 2566 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:56:03.257673 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.257660 2566 manager.go:341] "Starting Device Plugin manager" Oct 30 06:56:03.257763 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.257693 2566 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:56:03.257763 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.257705 2566 server.go:87] "Starting device plugin registration server" Oct 30 06:56:03.257972 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.257959 2566 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:56:03.258026 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.257971 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:56:03.258079 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.258053 2566 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:56:03.258144 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.258130 2566 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:56:03.258189 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.258144 2566 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:56:03.258739 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.258718 2566 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:56:03.258803 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.258773 2566 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:03.316316 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.316146 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:56:03.318069 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.317534 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:56:03.318069 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.317560 2566 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:56:03.318069 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.317581 2566 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:56:03.318069 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.317622 2566 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 06:56:03.332954 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.332933 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:03.358126 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.358112 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.359002 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.358985 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.359083 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.359016 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.359083 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.359031 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.359083 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.359065 2566 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.378939 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.378921 2566 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.379007 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.378942 2566 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-90.ec2.internal\": node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:03.418559 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.418533 2566 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal"] Oct 30 06:56:03.418700 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.418587 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.419451 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.419434 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.419534 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.419465 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.419534 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.419480 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.421365 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.421350 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.421519 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.421504 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.421581 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.421539 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.422169 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422149 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.422258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422176 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.422258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422191 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.422258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422154 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.422258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422254 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.422397 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.422275 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.424046 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.424033 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.424090 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.424057 2566 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:56:03.424760 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.424746 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:56:03.424840 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.424769 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:56:03.424840 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.424781 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:56:03.425506 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.425490 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:03.452959 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.452940 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-90.ec2.internal\" not found" node="ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.460045 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:03.460027 2566 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-90.ec2.internal\" not found" node="ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.485268 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.485241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.485387 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.485273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:03.485387 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:03.485300 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.228062 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.228030 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.287983 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.287957 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.288102 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.287989 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.288102 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.288012 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.288102 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.288088 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.288216 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.288124 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.288216 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.288088 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.328770 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.328747 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.429372 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.429344 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.460566 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.460537 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.465425 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.465309 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:05.529794 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.529762 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.630345 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.630281 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.730773 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.730740 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.795275 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.795246 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 06:56:05.795703 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.795391 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 06:56:05.795703 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.795426 2566 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 06:56:05.831578 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.831540 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.910128 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.910044 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:04 +0000 UTC" deadline="2028-04-04 16:02:50.784517243 +0000 UTC" Oct 30 06:56:05.910128 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.910087 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12537h6m44.87443473s" Oct 30 06:56:05.932740 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:05.932709 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:05.970073 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:05.970038 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2aec6afc092b0a7711821d66a44d1c3.slice/crio-b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6 WatchSource:0}: Error finding container b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6: Status 404 returned error can't find the container with id b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6 Oct 30 06:56:05.970399 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:05.970378 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1648b9f3a1682e8c87ccb70263aa6652.slice/crio-a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5 WatchSource:0}: Error finding container a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5: Status 404 returned error can't find the container with id a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5 Oct 30 06:56:05.975511 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:05.975495 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:56:06.022447 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.022398 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerStarted","Data":"b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6"} Oct 30 06:56:06.023438 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.023417 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" event={"ID":"1648b9f3a1682e8c87ccb70263aa6652","Type":"ContainerStarted","Data":"a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5"} Oct 30 06:56:06.033551 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.033530 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:06.134028 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.133995 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:06.234697 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.234603 2566 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:56:06.266954 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.266917 2566 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:06.310197 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.310163 2566 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:06.385645 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.385605 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:56:06.406056 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.406027 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:06.407208 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.407186 2566 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:56:06.428920 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.428880 2566 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:56:06.846091 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.846066 2566 apiserver.go:52] "Watching apiserver" Oct 30 06:56:06.885506 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.885468 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 06:56:06.885711 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.885687 2566 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:56:06.886141 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.886102 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ff84l","openshift-multus/network-metrics-daemon-xgzmb","openshift-ovn-kubernetes/ovnkube-node-wrvj8","kube-system/konnectivity-agent-4js78","openshift-dns/node-resolver-wdkkw","openshift-multus/multus-additional-cni-plugins-kwcnj","openshift-network-operator/iptables-alerter-94b4f","openshift-cluster-node-tuning-operator/tuned-wzqxw","openshift-image-registry/node-ca-rhhvm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal","openshift-network-diagnostics/network-check-target-7lhl4","kube-system/global-pull-secret-syncer-5n2mb","kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k"] Oct 30 06:56:06.888945 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.888913 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:06.891077 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.891055 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:06.891189 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.891131 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:06.893849 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.893829 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.895929 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.895905 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:06.898391 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.898366 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:06.898486 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.898407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bbc\" (UniqueName: \"kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:06.898486 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.898465 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:06.898680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.898657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:06.901893 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.900750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:06.903100 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.903081 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:06.903624 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.903602 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:56:06.904070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.904052 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:06.904208 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.904190 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:06.904284 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.904244 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:56:06.905878 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.905861 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.908227 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.908210 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:06.908333 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.908275 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:06.910573 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.910515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.913132 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.913113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.915395 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.915367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:06.915492 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:06.915449 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:06.939740 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.939716 2566 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:06.941443 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.941420 2566 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 06:56:06.941541 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.941491 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bnv8w" Oct 30 06:56:06.986085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.986056 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:56:06.998692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998650 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.998833 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw9hj\" (UniqueName: \"kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:06.998833 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.998926 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998838 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:06.998926 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.998926 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:06.998926 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2vt\" (UniqueName: \"kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.998988 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999028 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999049 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999064 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mwv7\" (UniqueName: \"kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999082 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999101 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtb4k\" (UniqueName: \"kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999160 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999178 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999225 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj55v\" (UniqueName: \"kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999255 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999278 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:06.999323 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999374 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999398 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999412 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999469 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999490 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzsb\" (UniqueName: \"kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999504 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999524 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999552 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69kj\" (UniqueName: \"kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:06.999814 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999683 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:06.999796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000267 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000353 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9qtb\" (UniqueName: \"kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000380 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000399 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.000421 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000489 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000521 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000557 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000645 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bbc\" (UniqueName: \"kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000671 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.000713 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000708 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000729 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000751 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000817 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000857 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000877 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.000977 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001000 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001025 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001048 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.001085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001087 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001133 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001153 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:07.001628 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.001267 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004266 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004276 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004289 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004291 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004346 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004353 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.004445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.004410 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005047 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005052 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005105 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005140 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005154 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005160 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005181 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005212 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005277 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005317 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005326 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005376 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005420 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005433 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005457 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005485 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005512 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005524 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005538 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005582 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:56:07.005692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005665 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:56:07.006513 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005707 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:56:07.006513 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.005733 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:56:07.008549 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.008454 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:56:07.008549 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.008519 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:56:07.022143 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.022117 2566 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:56:07.026246 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.026223 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bbc\" (UniqueName: \"kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.087832 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.087804 2566 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:56:07.101490 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.101490 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101452 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9hj\" (UniqueName: \"kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.101490 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101485 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101539 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101556 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101594 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2vt\" (UniqueName: \"kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101618 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101657 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwv7\" (UniqueName: \"kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.101748 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101789 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtb4k\" (UniqueName: \"kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101853 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101906 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101941 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj55v\" (UniqueName: \"kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.101986 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102015 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102052 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102083 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102106 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzsb\" (UniqueName: \"kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102233 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69kj\" (UniqueName: \"kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102273 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102292 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102337 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102352 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qtb\" (UniqueName: \"kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102401 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.102754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102421 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102451 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102467 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102489 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102513 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102608 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102624 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102723 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102739 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102763 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.103273 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102806 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102823 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102868 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102874 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102905 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102910 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102932 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102972 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.102991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.104034 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.103376 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103396 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103444 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.103454 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:07.603422076 +0000 UTC m=+3.206389156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103664 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103722 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103744 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103746 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103764 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103783 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103802 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103805 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103848 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103888 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103897 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103945 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.104666 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103974 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104259 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104322 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104331 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.104420 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.104457 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:07.604444016 +0000 UTC m=+3.207411082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104486 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104500 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104537 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104559 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104559 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104609 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104671 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104711 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104722 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104743 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.104969 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105017 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.105476 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105055 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105066 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105125 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105241 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105350 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105393 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105648 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105786 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.103075 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.106039 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.106149 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.106188 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.106242 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.106800 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.105400 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107048 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107255 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107450 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107484 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107533 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107563 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107570 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107779 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.107824 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107796 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.108211 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107908 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.108211 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.107992 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.117101 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.113164 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.117101 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.115914 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.117101 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.115989 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.117727 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.117701 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:07.126217 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.126182 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:07.126217 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.126214 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:07.126403 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.126230 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:07.126403 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.126303 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:07.626282312 +0000 UTC m=+3.229249399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:07.126523 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.126416 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzsb\" (UniqueName: \"kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.126523 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.126474 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69kj\" (UniqueName: \"kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:07.126627 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.126517 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9hj\" (UniqueName: \"kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.128308 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.128287 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwv7\" (UniqueName: \"kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.128422 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.128362 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj55v\" (UniqueName: \"kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.128866 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.128839 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtb4k\" (UniqueName: \"kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.129135 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.129117 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2vt\" (UniqueName: \"kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.131717 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.131595 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qtb\" (UniqueName: \"kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.204322 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.204288 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:56:07.217698 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.217672 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2a4773_14f8_467d_a46f_a05b38375900.slice/crio-697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652 WatchSource:0}: Error finding container 697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652: Status 404 returned error can't find the container with id 697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652 Oct 30 06:56:07.226832 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.226807 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:56:07.236216 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.236191 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:07.236324 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.236255 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:07.244311 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.244291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:56:07.262773 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.262739 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e60768_4e85_440c_8ddf_cb8f91fdd289.slice/crio-b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d WatchSource:0}: Error finding container b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d: Status 404 returned error can't find the container with id b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d Oct 30 06:56:07.267753 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.267731 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e4f6bc7_4192_4165_b16d_df0db49aedd9.slice/crio-ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a WatchSource:0}: Error finding container ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a: Status 404 returned error can't find the container with id ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a Oct 30 06:56:07.269184 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.268936 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87bd4fc_9395_46bf_87eb_13a159db993a.slice/crio-9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e WatchSource:0}: Error finding container 9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e: Status 404 returned error can't find the container with id 9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e Oct 30 06:56:07.272679 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.272654 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:56:07.273327 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.273303 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5300a292_d959_4d7b_a1fe_b1a1f81efd71.slice/crio-9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3 WatchSource:0}: Error finding container 9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3: Status 404 returned error can't find the container with id 9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3 Oct 30 06:56:07.274115 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.274054 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:56:07.276258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.276224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ff84l" Oct 30 06:56:07.278212 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.278158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:56:07.283436 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.283413 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679a9341_9428_4ae3_a14c_e72618082e3b.slice/crio-fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383 WatchSource:0}: Error finding container fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383: Status 404 returned error can't find the container with id fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383 Oct 30 06:56:07.288113 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.288017 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4538a8_12ff_45c9_ae2c_fcc65413dae1.slice/crio-c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b WatchSource:0}: Error finding container c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b: Status 404 returned error can't find the container with id c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b Oct 30 06:56:07.292197 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.292177 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b0503e_7393_44db_8f5c_1aaaf2043b07.slice/crio-f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de WatchSource:0}: Error finding container f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de: Status 404 returned error can't find the container with id f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de Oct 30 06:56:07.300869 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:07.300849 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3122922f_c624_4694_9e9d_17893f6d9af8.slice/crio-966c61d4431dcab9b98e9c36dc550969f969cefb11ddcac0b38200cf0c8ae63f WatchSource:0}: Error finding container 966c61d4431dcab9b98e9c36dc550969f969cefb11ddcac0b38200cf0c8ae63f: Status 404 returned error can't find the container with id 966c61d4431dcab9b98e9c36dc550969f969cefb11ddcac0b38200cf0c8ae63f Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.608831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.609038 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.609077 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.609134 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.609148 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.609130853 +0000 UTC m=+4.212097930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:07.609538 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.609183 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.609167889 +0000 UTC m=+4.212134968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:07.710052 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.710018 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:07.710231 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.710138 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:07.710231 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.710152 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:07.710231 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.710161 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:07.710231 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:07.710210 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:08.710198812 +0000 UTC m=+4.313165889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:07.945582 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.945472 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:06 +0000 UTC" deadline="2028-05-12 16:05:03.482462972 +0000 UTC" Oct 30 06:56:07.945582 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:07.945535 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13449h8m55.536931605s" Oct 30 06:56:08.020050 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.020016 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:08.020211 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.020149 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:08.044173 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.044099 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" event={"ID":"3122922f-c624-4694-9e9d-17893f6d9af8","Type":"ContainerStarted","Data":"966c61d4431dcab9b98e9c36dc550969f969cefb11ddcac0b38200cf0c8ae63f"} Oct 30 06:56:08.047706 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.047599 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerStarted","Data":"fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383"} Oct 30 06:56:08.052962 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.052859 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e"} Oct 30 06:56:08.061535 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.061468 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdkkw" event={"ID":"9e4f6bc7-4192-4165-b16d-df0db49aedd9","Type":"ContainerStarted","Data":"ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a"} Oct 30 06:56:08.066165 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.065937 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhhvm" event={"ID":"20e60768-4e85-440c-8ddf-cb8f91fdd289","Type":"ContainerStarted","Data":"b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d"} Oct 30 06:56:08.072659 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.072201 2566 generic.go:356] "Generic (PLEG): container finished" podID="e2aec6afc092b0a7711821d66a44d1c3" containerID="afc147c4329cbdff2bdd783d9a3ec4e45ac9e01edf5327268b6ebb8489d787fd" exitCode=0 Oct 30 06:56:08.072659 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.072284 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerDied","Data":"afc147c4329cbdff2bdd783d9a3ec4e45ac9e01edf5327268b6ebb8489d787fd"} Oct 30 06:56:08.081793 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.081690 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" event={"ID":"f0b0503e-7393-44db-8f5c-1aaaf2043b07","Type":"ContainerStarted","Data":"f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de"} Oct 30 06:56:08.085200 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.085092 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ff84l" event={"ID":"8d4538a8-12ff-45c9-ae2c-fcc65413dae1","Type":"ContainerStarted","Data":"c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b"} Oct 30 06:56:08.090899 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.090873 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4js78" event={"ID":"5300a292-d959-4d7b-a1fe-b1a1f81efd71","Type":"ContainerStarted","Data":"9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3"} Oct 30 06:56:08.107412 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.107381 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-94b4f" event={"ID":"8b2a4773-14f8-467d-a46f-a05b38375900","Type":"ContainerStarted","Data":"697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652"} Oct 30 06:56:08.115206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.115177 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" event={"ID":"1648b9f3a1682e8c87ccb70263aa6652","Type":"ContainerStarted","Data":"d7f14e2b8621412b53022697004eb94b1d9021b28ac7b7eef1cf56e76335bc79"} Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.618227 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.618296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.618422 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.618482 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.618463431 +0000 UTC m=+6.221430510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.618885 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:08.618968 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.618932 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.618918767 +0000 UTC m=+6.221885846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:08.719519 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.719478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:08.719795 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.719698 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:08.719795 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.719719 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:08.719795 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.719732 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:08.719795 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:08.719793 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:10.71977439 +0000 UTC m=+6.322741459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:08.946701 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.946655 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:06 +0000 UTC" deadline="2028-08-10 21:05:20.064035244 +0000 UTC" Oct 30 06:56:08.946701 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:08.946696 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15614h9m11.11734384s" Oct 30 06:56:09.023115 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:09.023083 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:09.023285 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:09.023224 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:09.023625 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:09.023605 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:09.023855 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:09.023831 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:09.131513 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:09.131475 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerStarted","Data":"52148b2c8f8f59bd2bf481511b77232bf648ba230cdacf6111fdcebb758b62c8"} Oct 30 06:56:09.170366 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:09.170262 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" podStartSLOduration=3.170243758 podStartE2EDuration="3.170243758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:08.154219804 +0000 UTC m=+3.757186891" watchObservedRunningTime="2026-10-30 06:56:09.170243758 +0000 UTC m=+4.773210845" Oct 30 06:56:10.019870 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:10.019839 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:10.020303 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.019996 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:10.633941 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:10.633894 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:10.634129 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:10.633963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:10.634188 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.634126 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:10.634188 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.634187 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:14.634169807 +0000 UTC m=+10.237136875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:10.634589 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.634567 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:10.634704 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.634644 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:14.63461414 +0000 UTC m=+10.237581214 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:10.735263 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:10.735226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:10.735430 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.735403 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:10.735430 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.735424 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:10.735549 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.735438 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:10.735549 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:10.735496 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:14.735479266 +0000 UTC m=+10.338446351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:11.020140 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:11.020068 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:11.020140 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:11.020114 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:11.020659 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:11.020214 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:11.020659 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:11.020505 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:11.151405 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:11.151173 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="70d62cb267998c80d168a6539c9ba4ef9b27f891043657da5c3522256efd31c3" exitCode=0 Oct 30 06:56:11.151405 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:11.151231 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"70d62cb267998c80d168a6539c9ba4ef9b27f891043657da5c3522256efd31c3"} Oct 30 06:56:11.227985 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:11.227474 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" podStartSLOduration=5.227460041 podStartE2EDuration="5.227460041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:09.170088797 +0000 UTC m=+4.773055884" watchObservedRunningTime="2026-10-30 06:56:11.227460041 +0000 UTC m=+6.830427122" Oct 30 06:56:12.020378 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:12.020295 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:12.020883 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:12.020429 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:12.167979 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:12.167193 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4js78" event={"ID":"5300a292-d959-4d7b-a1fe-b1a1f81efd71","Type":"ContainerStarted","Data":"08414db86c20b0bc954b59ba23ccbaa90da74c6a93c2439325b10082ec358264"} Oct 30 06:56:12.173208 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:12.172745 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhhvm" event={"ID":"20e60768-4e85-440c-8ddf-cb8f91fdd289","Type":"ContainerStarted","Data":"56bfe8631eeb3b1a347ffb489205c5cad23d9ddee7c14148a3c24e06aea45f8e"} Oct 30 06:56:12.246432 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:12.246369 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4js78" podStartSLOduration=3.332674776 podStartE2EDuration="7.246350388s" totalImagesPullingTime="3.913675612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:12.207907796 +0000 UTC m=+7.810874884" watchObservedRunningTime="2026-10-30 06:56:12.246350388 +0000 UTC m=+7.849317474" Oct 30 06:56:12.246872 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:12.246832 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rhhvm" podStartSLOduration=2.761199602 podStartE2EDuration="7.246818277s" totalImagesPullingTime="4.485618675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:12.24605725 +0000 UTC m=+7.849024336" watchObservedRunningTime="2026-10-30 06:56:12.246818277 +0000 UTC m=+7.849785365" Oct 30 06:56:13.022665 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:13.021751 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:13.022665 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:13.021889 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:13.022665 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:13.022316 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:13.022665 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:13.022392 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:13.177898 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:13.177864 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" event={"ID":"f0b0503e-7393-44db-8f5c-1aaaf2043b07","Type":"ContainerStarted","Data":"a1be005a4294f541c9f689494c9fa74616aebdade578566d99e683e5ab2817f8"} Oct 30 06:56:14.022597 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.022564 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:14.022802 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.022749 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:14.075924 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.075899 2566 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:56:14.188071 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.187942 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" event={"ID":"f0b0503e-7393-44db-8f5c-1aaaf2043b07","Type":"ContainerStarted","Data":"d903a6a589ae6de084b2c6adc68e68d60a522177e3db716347c7104638cba890"} Oct 30 06:56:14.230644 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.229913 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:14.231399 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.231000 2566 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.675384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.675459 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.675599 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.675675 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:22.67565752 +0000 UTC m=+18.278624599 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.676013 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:14.676094 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.676057 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:22.676043623 +0000 UTC m=+18.279010709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:14.777235 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.776617 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:14.777235 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.776812 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:14.777235 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.776832 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:14.777235 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.776841 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:14.777235 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:14.776892 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:22.77687593 +0000 UTC m=+18.379843016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:14.965003 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.964762 2566 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:56:14.075916165Z","UUID":"6c16a546-dbba-4622-8dd7-246143c8b056","Handler":null,"Name":"","Endpoint":""} Oct 30 06:56:14.967226 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.966837 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:56:14.967226 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:14.966860 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:56:15.023743 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.023490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:15.024081 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:15.023878 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:15.024348 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.024267 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:15.024392 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:15.024362 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:15.193134 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.193096 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wdkkw" event={"ID":"9e4f6bc7-4192-4165-b16d-df0db49aedd9","Type":"ContainerStarted","Data":"cd3c6916987762dbee83f32ed668b3bfd74a0f408c71e8a8280f3125bdb76eb3"} Oct 30 06:56:15.196134 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.196108 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ff84l" event={"ID":"8d4538a8-12ff-45c9-ae2c-fcc65413dae1","Type":"ContainerStarted","Data":"45fce66081c7ef8e0b6ad75604ca75ffa0f75fcc3cff92676d2d53be9880ae83"} Oct 30 06:56:15.232361 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.232269 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wdkkw" podStartSLOduration=2.506366995 podStartE2EDuration="10.232255079s" totalImagesPullingTime="7.725888084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:15.231708956 +0000 UTC m=+10.834676076" watchObservedRunningTime="2026-10-30 06:56:15.232255079 +0000 UTC m=+10.835222164" Oct 30 06:56:15.274937 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:15.273728 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ff84l" podStartSLOduration=3.032439725 podStartE2EDuration="10.273707114s" totalImagesPullingTime="7.241267389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:15.272736826 +0000 UTC m=+10.875703920" watchObservedRunningTime="2026-10-30 06:56:15.273707114 +0000 UTC m=+10.876674202" Oct 30 06:56:16.020646 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:16.020553 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:16.020801 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:16.020716 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:16.199505 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:16.199470 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="5023f8531c16e2f37efb8bf04fa4c3533a426178051efc8f5b2874bc8a0389ed" exitCode=0 Oct 30 06:56:16.200085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:16.199564 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"5023f8531c16e2f37efb8bf04fa4c3533a426178051efc8f5b2874bc8a0389ed"} Oct 30 06:56:16.203732 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:16.203684 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" event={"ID":"f0b0503e-7393-44db-8f5c-1aaaf2043b07","Type":"ContainerStarted","Data":"30bdf6ad8c6c4e1427ef7c7d2ffea727ce3e1c3931d25c55792bb3578c2e4ab4"} Oct 30 06:56:16.311880 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:16.311817 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" podStartSLOduration=3.718111439 podStartE2EDuration="11.311798361s" totalImagesPullingTime="7.593686922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:16.311224916 +0000 UTC m=+11.914192001" watchObservedRunningTime="2026-10-30 06:56:16.311798361 +0000 UTC m=+11.914765447" Oct 30 06:56:17.021575 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.021545 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:17.021726 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:17.021702 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:17.021853 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.021771 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:17.021853 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:17.021842 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:17.207659 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.207474 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" event={"ID":"3122922f-c624-4694-9e9d-17893f6d9af8","Type":"ContainerStarted","Data":"905d23e3fda238a7f6e2b90552b19abd680f710c34242f14dc4584b109e92aa6"} Oct 30 06:56:17.209709 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.209687 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="2809d178eb1f93c6431400a907bfdfe73fff9a187e491c0c60b3650d1662f751" exitCode=0 Oct 30 06:56:17.209818 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.209774 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"2809d178eb1f93c6431400a907bfdfe73fff9a187e491c0c60b3650d1662f751"} Oct 30 06:56:17.211448 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.211421 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-94b4f" event={"ID":"8b2a4773-14f8-467d-a46f-a05b38375900","Type":"ContainerStarted","Data":"a7ef1a2442d3b213a91b8858f54fc7523c34e60ea8afa32d8f0fae7b43ca7a08"} Oct 30 06:56:17.249378 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.249325 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" podStartSLOduration=2.646888782 podStartE2EDuration="12.249311855s" totalImagesPullingTime="9.602423073s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:17.248728708 +0000 UTC m=+12.851695795" watchObservedRunningTime="2026-10-30 06:56:17.249311855 +0000 UTC m=+12.852279005" Oct 30 06:56:17.332370 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:17.332320 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-94b4f" podStartSLOduration=4.452769371 podStartE2EDuration="12.332305343s" totalImagesPullingTime="7.879535972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:17.332098847 +0000 UTC m=+12.935065933" watchObservedRunningTime="2026-10-30 06:56:17.332305343 +0000 UTC m=+12.935272429" Oct 30 06:56:18.020995 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:18.020705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:18.021137 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:18.021024 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:18.215896 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:18.215862 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="023d64566d1c9c9bea5b4959380648a2886fb06a2bc919b9f5a216925512b134" exitCode=0 Oct 30 06:56:18.216338 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:18.215956 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"023d64566d1c9c9bea5b4959380648a2886fb06a2bc919b9f5a216925512b134"} Oct 30 06:56:19.022149 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:19.022116 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:19.022333 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:19.022261 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:19.022593 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:19.022571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:19.022746 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:19.022721 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:20.020539 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:20.020515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:20.020970 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:20.020620 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:20.222738 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:20.222650 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="d3acbfdbb611c5e8109c554d3ad4ab3f735e4be6165908c24928c51daf241725" exitCode=0 Oct 30 06:56:20.222738 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:20.222669 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"d3acbfdbb611c5e8109c554d3ad4ab3f735e4be6165908c24928c51daf241725"} Oct 30 06:56:21.020743 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:21.020707 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:21.021208 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:21.020756 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:21.021208 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:21.020846 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:21.021208 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:21.020942 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:21.228176 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:21.228137 2566 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="f88bd37620a2d217713c85abbb87a027b78af639b0cbe288177f4e8827b93c44" exitCode=0 Oct 30 06:56:21.228341 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:21.228183 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerDied","Data":"f88bd37620a2d217713c85abbb87a027b78af639b0cbe288177f4e8827b93c44"} Oct 30 06:56:22.020497 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.020314 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:22.020717 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.020586 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:22.141791 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.141751 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:22.142359 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.141897 2566 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:56:22.142359 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.142271 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:56:22.233370 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.233294 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" event={"ID":"679a9341-9428-4ae3-a14c-e72618082e3b","Type":"ContainerStarted","Data":"f232367028f0ffbfa293b9dd1b15fecd9fbf4dcea1809a84bcf394a77973fe44"} Oct 30 06:56:22.236282 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236256 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"d4130d2458441a79def5192be187b3fd0a5d7f3f3849c33cee23c0508de9cbba"} Oct 30 06:56:22.236401 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236316 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"3318ad87f527ffa0013a6ed6863fdbb007cbbb8862cc6dc21d6410551d37e840"} Oct 30 06:56:22.236401 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236348 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"f81548e453bb517e9e3db6131e120771e8b9131b59d19311ce4b5a11ee84ab8d"} Oct 30 06:56:22.236401 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236378 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"25848e0860e7fb7a5b7f2c854a61ad6581c55a4063a004d40d27b38f6ee4dbf3"} Oct 30 06:56:22.236521 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236409 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"1f9db7590426fa7bed1521de0261d08cae7e3c925954c3462350341059ae46f6"} Oct 30 06:56:22.236521 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.236441 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"095cecf07ebf408d39033a51a1947e1ffc650b4dac0b8ef13c071fc184549fb5"} Oct 30 06:56:22.280693 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.280648 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kwcnj" podStartSLOduration=7.422662047 podStartE2EDuration="17.280620505s" totalImagesPullingTime="9.857958458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:22.280006881 +0000 UTC m=+17.882973982" watchObservedRunningTime="2026-10-30 06:56:22.280620505 +0000 UTC m=+17.883587590" Oct 30 06:56:22.742551 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.742515 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:22.742747 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.742564 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:22.742747 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.742684 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:22.742747 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.742685 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:22.742747 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.742738 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.742722307 +0000 UTC m=+34.345689391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:22.742877 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.742750 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.74274465 +0000 UTC m=+34.345711717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:22.843133 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:22.843093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:22.843300 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.843244 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:22.843300 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.843264 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:22.843300 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.843274 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:22.843404 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:22.843321 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:56:38.843309427 +0000 UTC m=+34.446276495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:23.020284 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:23.020209 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:23.020284 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:23.020238 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:23.020465 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:23.020315 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:23.020465 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:23.020452 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:23.035393 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:23.035359 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:23.197416 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:23.197384 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:24.020313 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:24.020290 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:24.020412 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:24.020383 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:24.243728 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:24.243695 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"6799b8e9a267d25c1d1ed12ece4154dcfd109cb1cf7dd9241be26cad83596b30"} Oct 30 06:56:25.021557 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:25.021524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:25.021741 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:25.021663 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:25.021741 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:25.021692 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:25.021852 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:25.021787 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:26.019739 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.019711 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:26.020145 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:26.019796 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:26.255447 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.255412 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" event={"ID":"a87bd4fc-9395-46bf-87eb-13a159db993a","Type":"ContainerStarted","Data":"46a51097c4850213645ac33bf216eaab12d6ad877c22c963613c2bbaa4dbed28"} Oct 30 06:56:26.255793 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.255772 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:26.271839 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.271745 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:26.345844 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.345789 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" podStartSLOduration=6.951700259 podStartE2EDuration="21.345776122s" totalImagesPullingTime="14.394075863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:26.345538194 +0000 UTC m=+21.948505292" watchObservedRunningTime="2026-10-30 06:56:26.345776122 +0000 UTC m=+21.948743206" Oct 30 06:56:26.881252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.881209 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:26.991381 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:26.991348 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:27.020241 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:27.020207 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:27.020818 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:27.020207 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:27.020818 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:27.020340 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:27.020818 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:27.020452 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:27.257327 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:27.257251 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:27.257327 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:27.257324 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:27.271571 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:27.271549 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:28.020489 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.020462 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:28.020986 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:28.020591 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:28.238213 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.238118 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n2mb"] Oct 30 06:56:28.238353 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.238247 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:28.238422 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:28.238364 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:28.238764 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.238748 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xgzmb"] Oct 30 06:56:28.238839 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.238822 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:28.238909 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:28.238895 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:28.239222 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.239203 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lhl4"] Oct 30 06:56:28.258843 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.258817 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:28.258956 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:28.258895 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:28.424004 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.423969 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:28.533561 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:28.533530 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:29.814993 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:29.814959 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:29.949946 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:29.949918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:30.020685 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.020654 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:30.020685 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.020677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:30.020880 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.020681 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:30.020880 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:30.020749 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:30.020880 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:30.020829 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:30.020979 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:30.020900 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:30.090072 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.090000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:30.288021 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.287984 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:30.420876 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.420796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:30.536105 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:30.536074 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:31.126095 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:31.126064 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:32.020524 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:32.020495 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:32.020811 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:32.020552 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:32.020811 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:32.020566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:32.020811 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:32.020672 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:32.020811 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:32.020710 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:32.020811 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:32.020801 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:33.595502 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:33.595468 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:34.020338 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.020121 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:34.020540 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.020156 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:34.020540 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:34.020386 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:34.020540 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:34.020439 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:34.020540 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.020163 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:34.020707 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:34.020571 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:34.185174 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.185144 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:34.298947 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.298918 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:34.648560 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.648490 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:34.888043 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:34.888009 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:35.015720 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:35.015626 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:35.125596 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:35.125563 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:35.356312 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:35.356284 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:35.502255 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:35.502222 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:35.624927 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:35.624844 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:36.019956 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.019868 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:36.020406 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.019969 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:36.020406 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.020002 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:36.020406 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:36.019972 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:36.020406 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:36.020094 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:36.020406 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:36.020168 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:36.519594 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.519556 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:36.745199 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.745163 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:36.926289 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:36.926232 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:37.125871 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:37.125832 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:37.346247 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:37.346216 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:37.468087 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:37.468056 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:38.020433 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.020405 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:38.020433 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.020419 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:38.020680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.020502 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:38.020680 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.020504 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:38.020680 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.020568 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:38.020680 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.020625 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:38.559425 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.559396 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:38.724292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.724260 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:38.757339 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.757308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:38.757441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.757362 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:38.757478 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.757459 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:38.757523 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.757514 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret podName:6ca9fe1e-35ee-469e-b4af-7c13480101f7 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:10.757501343 +0000 UTC m=+66.360468410 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret") pod "global-pull-secret-syncer-5n2mb" (UID: "6ca9fe1e-35ee-469e-b4af-7c13480101f7") : object "kube-system"/"original-pull-secret" not registered Oct 30 06:56:38.757571 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.757460 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:38.757611 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.757581 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs podName:77bfd548-5933-4191-a2b2-a9ec19692a83 nodeName:}" failed. No retries permitted until 2026-10-30 06:57:10.757567914 +0000 UTC m=+66.360534991 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs") pod "network-metrics-daemon-xgzmb" (UID: "77bfd548-5933-4191-a2b2-a9ec19692a83") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 06:56:38.857727 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.857614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:38.857868 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.857770 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 06:56:38.857868 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.857790 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 06:56:38.857868 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.857800 2566 projected.go:196] Error preparing data for projected volume kube-api-access-6w2qb for pod openshift-network-diagnostics/network-check-target-7lhl4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:38.857868 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:38.857845 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb podName:869e41d6-ed18-44a7-96ec-74efb259f79f nodeName:}" failed. No retries permitted until 2026-10-30 06:57:10.857833206 +0000 UTC m=+66.460800274 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6w2qb" (UniqueName: "kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb") pod "network-check-target-7lhl4" (UID: "869e41d6-ed18-44a7-96ec-74efb259f79f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 06:56:38.921059 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:38.921020 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:39.041925 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:39.041898 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdkkw_9e4f6bc7-4192-4165-b16d-df0db49aedd9/dns-node-resolver/0.log" Oct 30 06:56:39.122062 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:39.121967 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:39.314766 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:39.314734 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:40.019891 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.019854 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:40.020292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.019918 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:40.020292 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.020008 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:40.020292 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:40.020021 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:40.020292 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:40.020082 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:40.020292 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:40.020164 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:40.242190 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.242163 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhhvm_20e60768-4e85-440c-8ddf-cb8f91fdd289/node-ca/0.log" Oct 30 06:56:40.519358 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.519326 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:40.630872 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.630837 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:40.743892 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.743860 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:40.922594 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:40.922556 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:41.122650 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:41.122610 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:41.323543 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:41.323509 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.020368 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.020332 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:42.020368 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.020362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:42.020613 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.020367 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:42.020613 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:42.020421 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:42.020613 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:42.020490 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:42.020613 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:42.020556 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:42.516265 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.516233 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.719058 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.719026 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.838688 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.838659 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.955932 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:42.955900 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:43.121407 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:43.121320 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:43.327090 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:43.327060 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:43.440595 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:43.440511 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:44.020261 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.020211 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:44.020261 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.020260 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:44.020726 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.020316 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:44.020726 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:44.020332 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:44.020726 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:44.020393 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:44.020726 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:44.020437 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:44.520933 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.520894 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:44.718349 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.718305 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:44.914518 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:44.914483 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:45.115654 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:45.115593 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:45.231970 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:45.231891 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:45.371049 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:45.371015 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:45.555554 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:45.555523 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:45.734074 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:45.734043 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:46.020785 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:46.020711 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:46.020942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:46.020711 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:46.020942 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:46.020813 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:46.020942 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:46.020878 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:46.020942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:46.020717 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:46.020942 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:46.020941 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:48.020271 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:48.020232 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:48.020767 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:48.020235 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:48.020767 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:48.020351 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:48.020767 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:48.020232 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:48.020767 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:48.020415 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:48.020767 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:48.020540 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:50.020770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.020733 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:50.021171 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.020810 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:50.021171 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.020837 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:50.021171 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:50.020916 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xgzmb" podUID="77bfd548-5933-4191-a2b2-a9ec19692a83" Oct 30 06:56:50.021171 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:50.020954 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7lhl4" podUID="869e41d6-ed18-44a7-96ec-74efb259f79f" Oct 30 06:56:50.021171 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:56:50.021076 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5n2mb" podUID="6ca9fe1e-35ee-469e-b4af-7c13480101f7" Oct 30 06:56:50.190616 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.190582 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:50.385858 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.385823 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:50.508536 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.508504 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:50.616414 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:50.616382 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:51.197115 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.197044 2566 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeReady" Oct 30 06:56:51.197461 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.197165 2566 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 06:56:51.256435 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.256404 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-55nz9"] Oct 30 06:56:51.258240 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.258226 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.262869 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.262847 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mwst9"] Oct 30 06:56:51.264567 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.264549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.272983 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.272959 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:56:51.272983 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.272977 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:56:51.273134 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.273001 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:56:51.280520 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.280497 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55nz9"] Oct 30 06:56:51.283523 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.283503 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sm2hk"] Oct 30 06:56:51.289112 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.289088 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwst9"] Oct 30 06:56:51.289226 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.289193 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.310920 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.310891 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sm2hk"] Oct 30 06:56:51.375506 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375472 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:56:51.375687 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375624 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:56:51.375758 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375739 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:56:51.375758 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375743 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:56:51.375835 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375768 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:56:51.375835 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375778 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:56:51.375835 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375802 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:56:51.375926 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.375902 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:56:51.385766 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.385752 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:56:51.446186 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.446291 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446191 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.446291 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446221 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.446404 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbkd2\" (UniqueName: \"kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.446451 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446435 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.446498 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446486 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.446531 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbvw\" (UniqueName: \"kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.446531 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446525 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.446595 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446549 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.446652 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.446652 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.446629 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmk2\" (UniqueName: \"kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.547484 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547454 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.547585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmk2\" (UniqueName: \"kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.547585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547512 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.547585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547532 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.547585 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547559 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.547854 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547834 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkd2\" (UniqueName: \"kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.547942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547872 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.547942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.547942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbvw\" (UniqueName: \"kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.548086 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.548086 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.547976 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.548086 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.548006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.548086 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.548072 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.548305 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.548283 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.548371 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.548325 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.548567 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.548547 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.551413 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.551392 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.551486 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.551412 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.551486 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.551474 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.570520 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.570491 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmk2\" (UniqueName: \"kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.570684 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.570667 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbvw\" (UniqueName: \"kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.570761 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.570683 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkd2\" (UniqueName: \"kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.577490 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.577470 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:56:51.599072 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.599052 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:56:51.869728 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.869659 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:51.883881 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.883692 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mwst9"] Oct 30 06:56:51.885422 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:51.885393 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf080f672_8cc7_4b00_a975_14ea5c5b1ae8.slice/crio-128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af WatchSource:0}: Error finding container 128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af: Status 404 returned error can't find the container with id 128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af Oct 30 06:56:51.911560 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:51.910706 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sm2hk"] Oct 30 06:56:51.913017 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:51.912991 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a74e65_a10a_423a_b889_8105cdb97b1b.slice/crio-2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a WatchSource:0}: Error finding container 2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a: Status 404 returned error can't find the container with id 2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a Oct 30 06:56:52.020023 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.019991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:56:52.020178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.020069 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:56:52.020217 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.020183 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:56:52.035278 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035255 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:56:52.035432 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035364 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:56:52.035551 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035532 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:56:52.035598 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035533 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:56:52.035598 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035533 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:56:52.035598 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035551 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:56:52.035598 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.035589 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:56:52.165899 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.165878 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-55nz9"] Oct 30 06:56:52.167528 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:52.167498 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6215b7c5_298f_481a_b3ce_884ad7382b26.slice/crio-7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87 WatchSource:0}: Error finding container 7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87: Status 404 returned error can't find the container with id 7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87 Oct 30 06:56:52.314854 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.314820 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55nz9" event={"ID":"6215b7c5-298f-481a-b3ce-884ad7382b26","Type":"ContainerStarted","Data":"7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87"} Oct 30 06:56:52.316125 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.316089 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sm2hk" event={"ID":"01a74e65-a10a-423a-b889-8105cdb97b1b","Type":"ContainerStarted","Data":"2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a"} Oct 30 06:56:52.317751 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.317724 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwst9" event={"ID":"f080f672-8cc7-4b00-a975-14ea5c5b1ae8","Type":"ContainerStarted","Data":"b7381ec2eb8b0c2518a708f8284adc11f299f670abce67a45cdd8189308e0482"} Oct 30 06:56:52.317869 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.317779 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwst9" event={"ID":"f080f672-8cc7-4b00-a975-14ea5c5b1ae8","Type":"ContainerStarted","Data":"128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af"} Oct 30 06:56:52.656281 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.656233 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:52.775825 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.775756 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qd27n"] Oct 30 06:56:52.777805 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.777783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:52.781878 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.781807 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:52.794432 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794409 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 06:56:52.794602 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794583 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 06:56:52.794699 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794604 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ds4sf\"" Oct 30 06:56:52.794699 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794585 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 06:56:52.794699 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794584 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 06:56:52.794818 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.794712 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 06:56:52.807859 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.807826 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qd27n"] Oct 30 06:56:52.892830 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.892796 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:52.960391 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.960345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:52.960552 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.960413 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:52.960552 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.960453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:52.960552 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:52.960493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd6nw\" (UniqueName: \"kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.061683 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.061626 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6nw\" (UniqueName: \"kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.061874 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.061732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.061874 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.061779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.061874 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.061814 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.062611 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.062573 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.064923 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.064876 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.065026 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.065003 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.085885 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.085856 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6nw\" (UniqueName: \"kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.092584 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.092560 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:56:53.323285 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.323180 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwst9" event={"ID":"f080f672-8cc7-4b00-a975-14ea5c5b1ae8","Type":"ContainerStarted","Data":"1e3ba7bccd9a11387f469e9dab6ebafa5360201371bfefa48adf82ceb683f4d2"} Oct 30 06:56:53.390243 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.390213 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:53.404673 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.404627 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-qd27n"] Oct 30 06:56:53.408896 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:53.408854 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799484a3_d828_4795_b681_710baad6a32c.slice/crio-b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a WatchSource:0}: Error finding container b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a: Status 404 returned error can't find the container with id b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a Oct 30 06:56:53.527360 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:53.527293 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:54.328687 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.328625 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55nz9" event={"ID":"6215b7c5-298f-481a-b3ce-884ad7382b26","Type":"ContainerStarted","Data":"ce87bc1fd3408fecd4386158ba1a56151566b43cf1f5d75d77f6d42c0676108a"} Oct 30 06:56:54.329046 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.328717 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-55nz9" event={"ID":"6215b7c5-298f-481a-b3ce-884ad7382b26","Type":"ContainerStarted","Data":"7564e17f743e9d2081bb5310e9ce0fe4c777ab462033655f90099947a9b6b6d6"} Oct 30 06:56:54.329499 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.329478 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-55nz9" Oct 30 06:56:54.332076 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.332046 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mwst9" event={"ID":"f080f672-8cc7-4b00-a975-14ea5c5b1ae8","Type":"ContainerStarted","Data":"1f709ec43359315809c287cc85aa252e6981d7d7a72eff74cd07ea66d7045938"} Oct 30 06:56:54.333413 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.333390 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" event={"ID":"799484a3-d828-4795-b681-710baad6a32c","Type":"ContainerStarted","Data":"b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a"} Oct 30 06:56:54.371008 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.370958 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-55nz9" podStartSLOduration=1.5263353990000001 podStartE2EDuration="3.370943292s" totalImagesPullingTime="1.844607893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:54.370225123 +0000 UTC m=+49.973192217" watchObservedRunningTime="2026-10-30 06:56:54.370943292 +0000 UTC m=+49.973910374" Oct 30 06:56:54.413177 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:54.413114 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mwst9" podStartSLOduration=1.2436091280000001 podStartE2EDuration="3.413097102s" totalImagesPullingTime="2.169487974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:54.411934838 +0000 UTC m=+50.014901924" watchObservedRunningTime="2026-10-30 06:56:54.413097102 +0000 UTC m=+50.016064189" Oct 30 06:56:55.246856 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.246823 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.338028 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.337996 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" event={"ID":"799484a3-d828-4795-b681-710baad6a32c","Type":"ContainerStarted","Data":"b6290a45bcf47c7711612805bf87976088e2fb81c5c2c6d8efc8dff5f943bec7"} Oct 30 06:56:55.338427 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.338057 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" event={"ID":"799484a3-d828-4795-b681-710baad6a32c","Type":"ContainerStarted","Data":"bf9b4247d373357fbfdb8df49931380a0f21bb60ff4090718d22d43b89b086a9"} Oct 30 06:56:55.339524 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.339494 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sm2hk" event={"ID":"01a74e65-a10a-423a-b889-8105cdb97b1b","Type":"ContainerStarted","Data":"42ebe8c7d64d103d50e2492407848b383bf3d08dae77ecb51527e35a3b43f062"} Oct 30 06:56:55.355334 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.355307 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.501794 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.501706 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.534538 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.534493 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sm2hk" podStartSLOduration=1.60797105 podStartE2EDuration="4.534478734s" totalImagesPullingTime="2.926507684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:55.533860291 +0000 UTC m=+51.136827376" watchObservedRunningTime="2026-10-30 06:56:55.534478734 +0000 UTC m=+51.137445818" Oct 30 06:56:55.577002 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.576935 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" podStartSLOduration=2.289265184 podStartE2EDuration="3.576906092s" totalImagesPullingTime="1.287640908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:55.575506091 +0000 UTC m=+51.178473175" watchObservedRunningTime="2026-10-30 06:56:55.576906092 +0000 UTC m=+51.179873177" Oct 30 06:56:55.613627 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.613586 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.729253 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:55.729216 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:56.603590 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:56.603561 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:56.822901 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:56.822863 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.003985 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.003909 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.125251 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.125224 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.243336 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.243301 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.378244 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.378208 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.416004 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.415975 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs"] Oct 30 06:56:57.418253 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.418238 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.438297 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.438270 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z2tvh\"" Oct 30 06:56:57.450629 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.450606 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 06:56:57.450768 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.450663 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 06:56:57.452646 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.452619 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs"] Oct 30 06:56:57.469521 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.469496 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg"] Oct 30 06:56:57.471521 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.471506 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.485192 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.485171 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-82xqd"] Oct 30 06:56:57.487020 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.487005 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.494891 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.494872 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 06:56:57.499092 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gvn9\" (UniqueName: \"kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499176 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499107 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499176 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499143 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t8lp\" (UniqueName: \"kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499286 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499180 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499286 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.499286 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499259 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499390 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499390 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499454 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499396 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.499454 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499544 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499466 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499544 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499485 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499657 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499556 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499657 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wv2r\" (UniqueName: \"kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.499746 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499731 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499789 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.499821 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499802 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.499821 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499818 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.499901 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.499836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.517970 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.517951 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg"] Oct 30 06:56:57.533251 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.533221 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.534178 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534164 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 06:56:57.534720 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534696 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 06:56:57.534812 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534741 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 06:56:57.534896 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534882 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 06:56:57.534938 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534900 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gbbvr\"" Oct 30 06:56:57.534938 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.534927 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pnvp5\"" Oct 30 06:56:57.588375 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.588345 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 06:56:57.600603 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.600736 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.600736 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600626 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.600736 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600666 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvn9\" (UniqueName: \"kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.600736 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.600910 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8lp\" (UniqueName: \"kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.600965 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600932 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601017 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.601017 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.600995 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601119 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601119 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.601119 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.601258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601138 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.601258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601179 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601258 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv2r\" (UniqueName: \"kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.601441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.601441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601341 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601359 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601441 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601412 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601703 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601466 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.601703 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601645 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601806 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601706 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.601889 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601869 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.601971 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.601953 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.602153 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.602128 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.602228 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.602134 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.602776 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.602745 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.603768 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.603744 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.604091 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.603795 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.604091 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.603811 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.604380 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.604349 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.604459 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.604352 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.604653 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.604617 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.623183 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.623155 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvn9\" (UniqueName: \"kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.625104 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.625081 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv2r\" (UniqueName: \"kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.625241 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.625221 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8lp\" (UniqueName: \"kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.658327 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.658254 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.729414 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.729383 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:56:57.783072 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.783039 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:56:57.799018 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.798398 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:56:57.810998 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:57.810951 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec28222_b8cd_4a03_8a22_3a0d81e5b50f.slice/crio-0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498 WatchSource:0}: Error finding container 0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498: Status 404 returned error can't find the container with id 0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498 Oct 30 06:56:57.864811 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.864783 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.975821 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:57.975740 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:58.033828 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.033806 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs"] Oct 30 06:56:58.035216 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:58.035192 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe05ba0_4fc7_4df3_bf3b_af4feadb519d.slice/crio-25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b WatchSource:0}: Error finding container 25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b: Status 404 returned error can't find the container with id 25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b Oct 30 06:56:58.090068 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.090040 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg"] Oct 30 06:56:58.091258 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:58.091230 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc44517_a592_4c5e_9a42_96047cabe380.slice/crio-9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0 WatchSource:0}: Error finding container 9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0: Status 404 returned error can't find the container with id 9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0 Oct 30 06:56:58.350067 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.350031 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82xqd" event={"ID":"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f","Type":"ContainerStarted","Data":"0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498"} Oct 30 06:56:58.351480 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.351442 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" event={"ID":"3fc44517-a592-4c5e-9a42-96047cabe380","Type":"ContainerStarted","Data":"9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0"} Oct 30 06:56:58.353365 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.353342 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" event={"ID":"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d","Type":"ContainerStarted","Data":"baa46b1817243f5a49f2eb149e8dceda56bc719c2867369bdce75007c5141930"} Oct 30 06:56:58.353463 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.353395 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" event={"ID":"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d","Type":"ContainerStarted","Data":"65040f04da0d17a068d81eb37270bb0e8e9f1c4d17e7f645f8dd5fae406987bf"} Oct 30 06:56:58.353463 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.353426 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" event={"ID":"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d","Type":"ContainerStarted","Data":"25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b"} Oct 30 06:56:58.541604 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.541566 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:58.656969 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.656916 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:58.830550 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.830515 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:58.836097 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.836058 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:56:58.841033 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.839243 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856250 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856251 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856326 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856335 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856341 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856355 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856387 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 06:56:58.856442 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856389 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 06:56:58.856902 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.856503 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cnr7r\"" Oct 30 06:56:58.869550 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.869519 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 06:56:58.870427 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.870405 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:56:58.914269 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914184 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914269 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914240 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914351 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914378 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914572 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914451 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914572 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914476 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914572 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914493 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914720 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914720 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914659 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914720 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914830 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914732 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:58.914830 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:58.914753 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015457 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015416 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015663 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015658 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015740 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015826 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.015961 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.015854 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.018770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.017834 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.018770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.017963 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.018770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.019045 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.018770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.019341 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.018770 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.019375 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.021134 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.021111 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.022547 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.022522 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.023430 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.023405 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.023677 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.023653 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.023677 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.023674 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.024120 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.024059 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.025247 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.025214 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.040528 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.040472 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.167733 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.167705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:56:59.272613 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.272582 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:56:59.358483 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.358416 2566 generic.go:356] "Generic (PLEG): container finished" podID="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" containerID="1b5fba996e3ab1818ab3a2320609ac4fe3d690b8cf6e52ec90e27ac2dbd927b7" exitCode=0 Oct 30 06:56:59.358483 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.358463 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82xqd" event={"ID":"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f","Type":"ContainerDied","Data":"1b5fba996e3ab1818ab3a2320609ac4fe3d690b8cf6e52ec90e27ac2dbd927b7"} Oct 30 06:56:59.360231 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.360195 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" event={"ID":"3fc44517-a592-4c5e-9a42-96047cabe380","Type":"ContainerStarted","Data":"65db16a687c3192adb0015bbcdb9f0a6a937971fb5c96ac625fef2cee4b7d316"} Oct 30 06:56:59.363176 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.362648 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" event={"ID":"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d","Type":"ContainerStarted","Data":"3451ecd046fbb0a99cb6825e12ecc4c9994cd8be2fbf34b3c1d5cd7cd0639512"} Oct 30 06:56:59.463742 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.463710 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:59.508069 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.508003 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" podStartSLOduration=1.533886767 podStartE2EDuration="2.507985371s" totalImagesPullingTime="974.098604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:56:59.444505238 +0000 UTC m=+55.047472324" watchObservedRunningTime="2026-10-30 06:56:59.507985371 +0000 UTC m=+55.110952456" Oct 30 06:56:59.508680 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.508661 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:56:59.513918 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:56:59.513879 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df7d331_9233_4871_9d2f_6961c18ca460.slice/crio-bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405 WatchSource:0}: Error finding container bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405: Status 404 returned error can't find the container with id bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405 Oct 30 06:56:59.593613 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.593584 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:59.644760 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.644728 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-58f5c47444-cfzwr"] Oct 30 06:56:59.648513 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.648491 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.665156 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665131 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 06:56:59.665156 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665131 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 06:56:59.665545 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665327 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 06:56:59.665545 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665344 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 06:56:59.665545 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665364 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lknhq\"" Oct 30 06:56:59.665545 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665401 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1jff2kns3ngd6\"" Oct 30 06:56:59.665545 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.665428 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 06:56:59.673565 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.673499 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58f5c47444-cfzwr"] Oct 30 06:56:59.722366 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722506 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722394 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn9sn\" (UniqueName: \"kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722506 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722573 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722513 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722573 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722734 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722665 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722788 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722765 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722835 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722799 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.722905 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.722888 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.823962 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.823922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.823971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9sn\" (UniqueName: \"kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824097 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824163 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824150 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824416 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824195 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824416 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824222 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.824867 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.824808 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.825085 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.825065 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827080 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827050 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827336 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827312 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827422 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827337 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827552 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827535 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827604 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827548 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.827886 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.827865 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.847250 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.847216 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9sn\" (UniqueName: \"kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:56:59.965136 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:56:59.965048 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:00.276560 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.276531 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-58f5c47444-cfzwr"] Oct 30 06:57:00.277822 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:00.277791 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c07bc5_c5be_4d09_9f96_1e50f37423be.slice/crio-78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03 WatchSource:0}: Error finding container 78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03: Status 404 returned error can't find the container with id 78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03 Oct 30 06:57:00.367552 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.367516 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405"} Oct 30 06:57:00.368859 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.368829 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03"} Oct 30 06:57:00.371512 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.371471 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82xqd" event={"ID":"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f","Type":"ContainerStarted","Data":"7e6d753496e5c3bdeb72f1f283299b1ca3339ae67ef87caa740e833507c33a98"} Oct 30 06:57:00.371667 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.371540 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-82xqd" event={"ID":"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f","Type":"ContainerStarted","Data":"f7a6ef8616cd654d66af3e315eef916fb282d12af2e36cf6d16a009e06e49e67"} Oct 30 06:57:00.378407 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.378374 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" event={"ID":"3fc44517-a592-4c5e-9a42-96047cabe380","Type":"ContainerStarted","Data":"7bc0cee4fd957669408789a8f5968afd702372381016a4678822531e157f3b2d"} Oct 30 06:57:00.378489 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.378449 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" event={"ID":"3fc44517-a592-4c5e-9a42-96047cabe380","Type":"ContainerStarted","Data":"153f77dc77ebd7a1071b2f143726e638b90205f47e0c5650ed693386105a1c7b"} Oct 30 06:57:00.419717 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.419660 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-82xqd" podStartSLOduration=2.516131295 podStartE2EDuration="3.419646508s" totalImagesPullingTime="821.193017ms" totalInitContainerRuntime="82.322196ms" isStatefulPod=true podCreationTimestamp="2026-10-30 06:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:00.41854821 +0000 UTC m=+56.021515298" watchObservedRunningTime="2026-10-30 06:57:00.419646508 +0000 UTC m=+56.022613584" Oct 30 06:57:00.461448 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.461397 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" podStartSLOduration=2.256033574 podStartE2EDuration="3.461386202s" totalImagesPullingTime="1.205352628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:00.461137258 +0000 UTC m=+56.064104344" watchObservedRunningTime="2026-10-30 06:57:00.461386202 +0000 UTC m=+56.064353287" Oct 30 06:57:00.689751 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.689724 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:00.818844 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.818801 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:00.981707 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:00.981618 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:01.382450 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:01.382416 2566 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e" exitCode=0 Oct 30 06:57:01.382651 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:01.382509 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e"} Oct 30 06:57:02.029522 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.029480 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-65bd9d49b8-7bvx4"] Oct 30 06:57:02.031577 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.031557 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.052940 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.052840 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 06:57:02.052940 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.052882 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 06:57:02.053212 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.053194 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 06:57:02.053293 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.053259 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 06:57:02.062361 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.062334 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s5mng\"" Oct 30 06:57:02.062521 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.062401 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9kbpgghqali20\"" Oct 30 06:57:02.066831 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.066801 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65bd9d49b8-7bvx4"] Oct 30 06:57:02.147145 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147316 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147152 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147316 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147316 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147268 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147316 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147304 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147515 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx82n\" (UniqueName: \"kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.147515 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.147370 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.247976 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248019 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248064 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248090 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx82n\" (UniqueName: \"kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248133 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248225 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248187 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.248596 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248568 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.249010 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.248986 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.249373 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.249355 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.249882 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.249859 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.250827 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.250806 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.252946 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.252920 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.253864 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.253843 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.255370 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.255344 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:02.256865 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.256843 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65fv7"] Oct 30 06:57:02.259467 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.259447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.272776 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.272747 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx82n\" (UniqueName: \"kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.279058 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.279027 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 06:57:02.279376 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.279314 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mzhb7\"" Oct 30 06:57:02.280944 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.280877 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65fv7"] Oct 30 06:57:02.345070 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.345045 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:02.350082 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.350053 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwv82\" (UniqueName: \"kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.350172 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.350102 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.398122 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.397847 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"d07cee209a8f46051d44a2c1761c7fda85f167245cfbd1b81e16676a0c0498ed"} Oct 30 06:57:02.398122 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.398010 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"0fdf39a531f51c99a6dae488859181fe6984dfbd3ab6333aec0647d3a840f4ba"} Oct 30 06:57:02.451099 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.451063 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwv82\" (UniqueName: \"kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.451274 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.451111 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.451443 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:57:02.451343 2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Oct 30 06:57:02.451443 ip-10-0-143-90 kubenswrapper[2566]: E1030 06:57:02.451407 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert podName:b1429333-ac51-49ee-8173-806b8ae7aeaf nodeName:}" failed. No retries permitted until 2026-10-30 06:57:02.95138896 +0000 UTC m=+58.554356039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert") pod "monitoring-plugin-77f4f76594-65fv7" (UID: "b1429333-ac51-49ee-8173-806b8ae7aeaf") : secret "monitoring-plugin-cert" not found Oct 30 06:57:02.477304 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.477267 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwv82\" (UniqueName: \"kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.544692 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.544599 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:02.662954 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.662928 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-65bd9d49b8-7bvx4"] Oct 30 06:57:02.664577 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:02.664542 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7f4421_c460_4690_b755_a6fef7f2e487.slice/crio-68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020 WatchSource:0}: Error finding container 68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020: Status 404 returned error can't find the container with id 68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020 Oct 30 06:57:02.956107 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.956034 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:02.958233 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:02.958214 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:03.213135 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.213071 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:03.281982 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.281952 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:03.393791 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.393769 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:03.404523 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.404491 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"b8f359541ce58a771aad8ee072c835bd9fa87e2401a74249f52fec8f6a8c375d"} Oct 30 06:57:03.405884 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.405855 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" event={"ID":"5a7f4421-c460-4690-b755-a6fef7f2e487","Type":"ContainerStarted","Data":"68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020"} Oct 30 06:57:03.409554 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.409531 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06"} Oct 30 06:57:03.409684 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.409590 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67"} Oct 30 06:57:03.409684 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.409627 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231"} Oct 30 06:57:03.409684 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.409676 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307"} Oct 30 06:57:03.409832 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.409700 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c"} Oct 30 06:57:03.520955 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:03.520932 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-65fv7"] Oct 30 06:57:03.525853 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:03.525819 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1429333_ac51_49ee_8173_806b8ae7aeaf.slice/crio-8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b WatchSource:0}: Error finding container 8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b: Status 404 returned error can't find the container with id 8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b Oct 30 06:57:04.417497 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.416744 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerStarted","Data":"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b"} Oct 30 06:57:04.419395 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.419366 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" event={"ID":"b1429333-ac51-49ee-8173-806b8ae7aeaf","Type":"ContainerStarted","Data":"8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b"} Oct 30 06:57:04.422832 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.422809 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"5a05fbe233b79102878a28ca8b20bde013ed8bee63b80485d948bf981e98a566"} Oct 30 06:57:04.422924 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.422880 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"101216350da0ab65b220c6f2fbdbc4fb89db7ea8b00ef9a1bac0d669cb2613e2"} Oct 30 06:57:04.422979 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.422923 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" event={"ID":"99c07bc5-c5be-4d09-9f96-1e50f37423be","Type":"ContainerStarted","Data":"5159d943eb3a34f10662a250016e466f1e3ef73d8048bc5ce99943221f5f5072"} Oct 30 06:57:04.422979 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.422967 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:04.424524 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.424494 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" event={"ID":"5a7f4421-c460-4690-b755-a6fef7f2e487","Type":"ContainerStarted","Data":"ffb25d1b2663ac05003cb835e364d13715d9502d7e7d5916cb82e69ea137db9c"} Oct 30 06:57:04.473533 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.473505 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:04.477453 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.477427 2566 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:04.482693 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.482626 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.501757 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.501717 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 06:57:04.503727 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.503686 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.543729448 podStartE2EDuration="6.50367305s" totalImagesPullingTime="2.949101822s" totalInitContainerRuntime="10.84178ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:04.502741622 +0000 UTC m=+60.105708709" watchObservedRunningTime="2026-10-30 06:57:04.50367305 +0000 UTC m=+60.106640134" Oct 30 06:57:04.513912 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.513886 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 06:57:04.514307 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.514133 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 06:57:04.514307 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.514201 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 06:57:04.514482 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.514245 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 06:57:04.526035 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526009 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 06:57:04.526167 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526097 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 06:57:04.526232 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526191 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 06:57:04.526285 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526236 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6pfvt\"" Oct 30 06:57:04.526336 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526299 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-66t9c9j0f6gnq\"" Oct 30 06:57:04.526336 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526323 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 06:57:04.526396 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.526369 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 06:57:04.527396 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.527377 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 06:57:04.527794 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.527779 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 06:57:04.528385 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.528364 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 06:57:04.545627 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.545597 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:04.572066 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572022 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572131 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572206 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572360 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572360 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572360 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572360 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572336 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572474 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572474 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572474 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572423 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572474 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572588 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572588 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572693 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572693 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572663 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572750 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.572750 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.572715 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.603670 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.603627 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 06:57:04.610837 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.610778 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" podStartSLOduration=1.043223411 podStartE2EDuration="2.610759264s" totalImagesPullingTime="1.567535853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:04.609545841 +0000 UTC m=+60.212512926" watchObservedRunningTime="2026-10-30 06:57:04.610759264 +0000 UTC m=+60.213726346" Oct 30 06:57:04.659744 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.658712 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" podStartSLOduration=2.835757682 podStartE2EDuration="5.658687075s" totalImagesPullingTime="2.822929393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:04.657005216 +0000 UTC m=+60.259972301" watchObservedRunningTime="2026-10-30 06:57:04.658687075 +0000 UTC m=+60.261654162" Oct 30 06:57:04.673659 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673820 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673820 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673820 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673732 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673820 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673845 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673874 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673914 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.673976 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.673971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674002 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674089 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674252 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674612 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674612 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674612 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674565 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674789 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674648 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674789 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674662 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.674789 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.674680 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.675328 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.675303 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.676203 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.675579 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.676203 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.676112 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.677895 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.677870 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.679697 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.679673 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.679806 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.679728 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.679863 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.679808 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.679863 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.679812 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.680296 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.680261 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.680368 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.680334 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.680481 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.680451 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.680548 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.680484 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.680548 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.680506 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.681555 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.681510 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.682445 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.682423 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.699460 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.699437 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:04.827695 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:04.827657 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:05.175004 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.174928 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:05.187865 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:05.187838 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6daadbca_3412_44d6_85ab_94e80b0ca09a.slice/crio-2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9 WatchSource:0}: Error finding container 2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9: Status 404 returned error can't find the container with id 2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9 Oct 30 06:57:05.344256 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.344229 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:05.408806 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.408762 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:05.429222 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.429145 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" event={"ID":"b1429333-ac51-49ee-8173-806b8ae7aeaf","Type":"ContainerStarted","Data":"ab78280cb041e24b97798c9cc9d6cedde41555bd3eb70e24a172856703149abc"} Oct 30 06:57:05.429222 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.429206 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:05.430912 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.430879 2566 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028" exitCode=0 Oct 30 06:57:05.431049 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.431017 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028"} Oct 30 06:57:05.431114 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.431084 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9"} Oct 30 06:57:05.435795 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.435771 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:05.475674 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.475605 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" podStartSLOduration=1.974018324 podStartE2EDuration="3.475593182s" totalImagesPullingTime="1.501574858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:05.474737673 +0000 UTC m=+61.077704755" watchObservedRunningTime="2026-10-30 06:57:05.475593182 +0000 UTC m=+61.078560267" Oct 30 06:57:05.559513 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:05.559486 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:07.476304 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:07.476255 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:07.594842 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:07.594810 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:09.450530 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450491 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e"} Oct 30 06:57:09.450953 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450575 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6"} Oct 30 06:57:09.450953 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450679 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de"} Oct 30 06:57:09.450953 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450728 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352"} Oct 30 06:57:09.450953 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450763 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b"} Oct 30 06:57:09.450953 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.450796 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerStarted","Data":"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888"} Oct 30 06:57:09.526466 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.526395 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.485997774 podStartE2EDuration="5.526373974s" totalImagesPullingTime="3.02305627s" totalInitContainerRuntime="17.31993ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:09.521734206 +0000 UTC m=+65.124701293" watchObservedRunningTime="2026-10-30 06:57:09.526373974 +0000 UTC m=+65.129341062" Oct 30 06:57:09.827807 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:09.827774 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:10.437710 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.437684 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:10.459653 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.459602 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:10.577399 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.577366 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:10.712201 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.712121 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:10.834942 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.834904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:10.835119 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.835055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:10.849120 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.849094 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:57:10.849668 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.849629 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:57:10.858152 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.858122 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:10.858259 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.858231 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:10.936122 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.936095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:10.944568 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.944541 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:57:10.949705 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.949687 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:57:10.951754 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.951737 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:10.961025 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.960999 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:57:10.963193 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.963136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:10.970387 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.970358 2566 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:57:10.980468 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:10.980448 2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:11.250050 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.249985 2566 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:57:11.257327 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.257256 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:11.257833 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.257801 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5n2mb"] Oct 30 06:57:11.259856 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:11.259832 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca9fe1e_35ee_469e_b4af_7c13480101f7.slice/crio-3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072 WatchSource:0}: Error finding container 3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072: Status 404 returned error can't find the container with id 3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072 Oct 30 06:57:11.275161 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.275137 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xgzmb"] Oct 30 06:57:11.276224 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:11.276189 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bfd548_5933_4191_a2b2_a9ec19692a83.slice/crio-fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0 WatchSource:0}: Error finding container fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0: Status 404 returned error can't find the container with id fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0 Oct 30 06:57:11.303888 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.303841 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:11.424062 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.424028 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:11.459503 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.459465 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgzmb" event={"ID":"77bfd548-5933-4191-a2b2-a9ec19692a83","Type":"ContainerStarted","Data":"fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0"} Oct 30 06:57:11.460649 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.460608 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5n2mb" event={"ID":"6ca9fe1e-35ee-469e-b4af-7c13480101f7","Type":"ContainerStarted","Data":"3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072"} Oct 30 06:57:11.571218 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:11.571189 2566 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7lhl4"] Oct 30 06:57:11.571948 ip-10-0-143-90 kubenswrapper[2566]: W1030 06:57:11.571923 2566 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869e41d6_ed18_44a7_96ec_74efb259f79f.slice/crio-b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215 WatchSource:0}: Error finding container b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215: Status 404 returned error can't find the container with id b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215 Oct 30 06:57:12.284382 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:12.284342 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:12.415900 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:12.415869 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:12.467708 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:12.467672 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lhl4" event={"ID":"869e41d6-ed18-44a7-96ec-74efb259f79f","Type":"ContainerStarted","Data":"b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215"} Oct 30 06:57:12.526408 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:12.526333 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.016131 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.016051 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.129264 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.129228 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.474422 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.474382 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgzmb" event={"ID":"77bfd548-5933-4191-a2b2-a9ec19692a83","Type":"ContainerStarted","Data":"a2ea1ea8ce2bb621698fb0c982e9a0f002b26b2cddf20e0df822b9468c846787"} Oct 30 06:57:13.474936 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.474448 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xgzmb" event={"ID":"77bfd548-5933-4191-a2b2-a9ec19692a83","Type":"ContainerStarted","Data":"e56ac37708dde57a2396cd3f0abac9cda8636d2f407451505cdfad1cb8a5e650"} Oct 30 06:57:13.517100 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.517046 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xgzmb" podStartSLOduration=67.397540143 podStartE2EDuration="1m8.517032703s" totalImagesPullingTime="1.11949256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:13.516181967 +0000 UTC m=+69.119149057" watchObservedRunningTime="2026-10-30 06:57:13.517032703 +0000 UTC m=+69.119999788" Oct 30 06:57:13.640482 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.640443 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.753302 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:13.753203 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:15.485136 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:15.485096 2566 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7lhl4" event={"ID":"869e41d6-ed18-44a7-96ec-74efb259f79f","Type":"ContainerStarted","Data":"ae6db2878a33f252a60a7654ffc52591cd5b2d735b37ce5cb330bd71be42adee"} Oct 30 06:57:15.485546 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:15.485207 2566 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:15.525018 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:15.524965 2566 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7lhl4" podStartSLOduration=67.384122109 podStartE2EDuration="1m10.524945448s" totalImagesPullingTime="3.140823339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:15.524299612 +0000 UTC m=+71.127266701" watchObservedRunningTime="2026-10-30 06:57:15.524945448 +0000 UTC m=+71.127912536" Oct 30 06:57:16.248031 ip-10-0-143-90 kubenswrapper[2566]: I1030 06:57:16.247813 2566 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:57:16.247974 ip-10-0-143-90 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 06:57:16.265082 ip-10-0-143-90 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 06:57:16.265248 ip-10-0-143-90 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 06:57:16.265295 ip-10-0-143-90 systemd[1]: kubelet.service: Consumed 4.739s CPU time. Oct 30 06:57:16.275371 ip-10-0-143-90 systemd[1]: Starting Kubernetes Kubelet... Oct 30 06:57:16.338567 ip-10-0-143-90 kubenswrapper[5894]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:57:16.338567 ip-10-0-143-90 kubenswrapper[5894]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 06:57:16.338567 ip-10-0-143-90 kubenswrapper[5894]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:57:16.338567 ip-10-0-143-90 kubenswrapper[5894]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 06:57:16.340985 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340891 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.340985 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340980 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.340985 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340986 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.340985 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340990 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340995 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.340999 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341003 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341008 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341012 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341016 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341019 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341023 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341026 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341030 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341035 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341038 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341042 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341049 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341054 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341063 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341067 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341071 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341075 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.341257 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341080 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341083 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341087 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341091 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341095 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341099 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341103 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341107 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341111 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341116 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341121 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341125 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341130 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341134 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341138 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341143 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341148 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341153 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341160 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.342152 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341166 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341173 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341178 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341183 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341188 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341194 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341199 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341203 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341207 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341213 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341217 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341222 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341226 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341231 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341235 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341239 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341243 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341247 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341251 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.343022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341255 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341259 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341263 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341268 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341272 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341277 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341281 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341285 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341289 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341294 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341298 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341303 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341307 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341312 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341317 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341322 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341326 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341331 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341335 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341339 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.343878 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341343 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341349 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341354 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341359 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341363 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341367 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341371 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341376 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341380 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341384 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341389 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341393 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341397 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341401 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341406 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341410 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341415 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341419 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341423 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.344785 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341428 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341432 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341436 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341440 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341444 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.341449 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342733 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342745 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342749 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342754 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342758 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342762 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342766 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342771 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342775 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342779 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342784 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342788 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342793 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.345698 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342797 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342802 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342806 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342811 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342816 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342820 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342824 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342828 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342833 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342837 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342842 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342846 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342851 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342856 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342860 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342864 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342869 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342874 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342879 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342884 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.346549 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342889 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342893 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342897 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342902 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342906 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342910 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342914 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342918 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342923 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342927 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342932 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342936 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342941 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342945 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342949 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342954 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342958 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342962 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342966 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.347413 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342970 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342975 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342979 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342984 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342988 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342992 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.342997 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343001 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343006 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343010 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343015 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343019 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343023 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343027 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343031 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343035 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343039 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343043 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343047 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343051 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.347952 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343056 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343065 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343070 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343074 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343078 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343083 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343087 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343092 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343097 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343101 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343106 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343110 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343114 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343120 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343125 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343129 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343133 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343139 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343147 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.348435 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343154 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343159 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343164 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343168 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343172 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343178 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343182 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343188 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343192 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343197 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343201 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343205 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343209 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343213 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.343219 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344114 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344124 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344129 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344133 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344138 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.348926 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344142 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344146 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344151 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344155 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344159 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344163 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344168 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344175 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344179 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344183 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344188 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344192 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344196 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344200 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344204 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344209 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344213 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344217 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344222 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344227 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.349464 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344232 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344236 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344241 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344245 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344249 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344254 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344261 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344268 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344273 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344277 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344282 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344286 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344290 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344294 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344298 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344303 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344307 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344312 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344316 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.350022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344322 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344326 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344331 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344335 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344339 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344343 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344347 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344351 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344357 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344361 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344365 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344370 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344374 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344381 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344385 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344390 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344394 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344399 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344403 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344407 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.350788 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344412 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344416 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344420 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344424 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344428 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344433 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344437 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344441 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344445 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344450 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344454 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344458 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344464 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344468 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344473 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344477 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344481 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344485 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344490 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.351352 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344494 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344498 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344502 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344506 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344510 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344515 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344519 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344524 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344529 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344533 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344537 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344541 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344546 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344550 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344554 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344558 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344565 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344571 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344575 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.351892 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344579 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344585 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344589 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.344593 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344762 5894 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344774 5894 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344790 5894 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344798 5894 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344804 5894 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344809 5894 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344816 5894 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344823 5894 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344829 5894 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344834 5894 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344839 5894 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344845 5894 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344850 5894 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344855 5894 flags.go:64] FLAG: --cgroup-root="" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344859 5894 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344864 5894 flags.go:64] FLAG: --client-ca-file="" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344869 5894 flags.go:64] FLAG: --cloud-provider="external" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344882 5894 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344895 5894 flags.go:64] FLAG: --cluster-domain="" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344899 5894 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 06:57:16.352442 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344905 5894 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344910 5894 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344916 5894 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344922 5894 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344927 5894 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344932 5894 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344937 5894 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344943 5894 flags.go:64] FLAG: --contention-profiling="false" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344948 5894 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344953 5894 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344959 5894 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344963 5894 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344970 5894 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344975 5894 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344982 5894 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344987 5894 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344992 5894 flags.go:64] FLAG: --enable-server="true" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.344997 5894 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345008 5894 flags.go:64] FLAG: --event-burst="100" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345013 5894 flags.go:64] FLAG: --event-qps="50" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345018 5894 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345023 5894 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345028 5894 flags.go:64] FLAG: --eviction-hard="" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345034 5894 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 06:57:16.353104 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345039 5894 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345043 5894 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345048 5894 flags.go:64] FLAG: --eviction-soft="" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345053 5894 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345058 5894 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345063 5894 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345069 5894 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345073 5894 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345078 5894 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345082 5894 flags.go:64] FLAG: --feature-gates="" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345093 5894 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345098 5894 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345103 5894 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345108 5894 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345113 5894 flags.go:64] FLAG: --healthz-port="10248" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345119 5894 flags.go:64] FLAG: --help="false" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345125 5894 flags.go:64] FLAG: --hostname-override="ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345130 5894 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345135 5894 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345139 5894 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345145 5894 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345151 5894 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345157 5894 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 06:57:16.353776 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345162 5894 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345167 5894 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345172 5894 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345177 5894 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345182 5894 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345186 5894 flags.go:64] FLAG: --kube-reserved="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345191 5894 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345195 5894 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345200 5894 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345205 5894 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345210 5894 flags.go:64] FLAG: --lock-file="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345216 5894 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345222 5894 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345227 5894 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345237 5894 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345242 5894 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345247 5894 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345252 5894 flags.go:64] FLAG: --logging-format="text" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345256 5894 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345262 5894 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345267 5894 flags.go:64] FLAG: --manifest-url="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345272 5894 flags.go:64] FLAG: --manifest-url-header="" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345278 5894 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345283 5894 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345292 5894 flags.go:64] FLAG: --max-pods="110" Oct 30 06:57:16.354390 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345297 5894 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345302 5894 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345307 5894 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345312 5894 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345317 5894 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345322 5894 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345328 5894 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345340 5894 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345345 5894 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345350 5894 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345355 5894 flags.go:64] FLAG: --pod-cidr="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345360 5894 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345364 5894 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345369 5894 flags.go:64] FLAG: --pods-per-core="0" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345374 5894 flags.go:64] FLAG: --port="10250" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345379 5894 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345384 5894 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b11a7f7464b1f6c4" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345389 5894 flags.go:64] FLAG: --qos-reserved="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345394 5894 flags.go:64] FLAG: --read-only-port="10255" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345399 5894 flags.go:64] FLAG: --register-node="true" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345404 5894 flags.go:64] FLAG: --register-with-taints="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345410 5894 flags.go:64] FLAG: --registry-burst="10" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345415 5894 flags.go:64] FLAG: --registry-qps="5" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345419 5894 flags.go:64] FLAG: --reserved-cpus="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345425 5894 flags.go:64] FLAG: --reserved-memory="" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345431 5894 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 06:57:16.355069 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345437 5894 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345442 5894 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345447 5894 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345452 5894 flags.go:64] FLAG: --runonce="false" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345456 5894 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345463 5894 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345468 5894 flags.go:64] FLAG: --seccomp-default="false" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345472 5894 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345477 5894 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345482 5894 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345487 5894 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345492 5894 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345499 5894 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345505 5894 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345509 5894 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345514 5894 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345519 5894 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345524 5894 flags.go:64] FLAG: --system-cgroups="" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345528 5894 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345537 5894 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345542 5894 flags.go:64] FLAG: --tls-cert-file="" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345546 5894 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345556 5894 flags.go:64] FLAG: --tls-min-version="" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345561 5894 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345566 5894 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 06:57:16.355737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345571 5894 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345575 5894 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345580 5894 flags.go:64] FLAG: --v="2" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345586 5894 flags.go:64] FLAG: --version="false" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345593 5894 flags.go:64] FLAG: --vmodule="" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345600 5894 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.345606 5894 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345861 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345870 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345875 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345879 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345884 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345891 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345895 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345899 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345904 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345908 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345913 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345917 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345923 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345928 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.356330 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345932 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345937 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345942 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345947 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345952 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345956 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345960 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345965 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345969 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345973 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345978 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345982 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345986 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345991 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.345995 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346000 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346004 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346008 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346014 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346018 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.356861 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346022 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346026 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346031 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346038 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346042 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346046 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346051 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346055 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346059 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346064 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346069 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346074 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346078 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346083 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346087 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346092 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346096 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346100 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346107 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.357496 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346114 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346119 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346125 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346130 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346134 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346138 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346143 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346147 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346151 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346155 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346160 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346164 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346169 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346174 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346178 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346182 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346188 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346192 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346196 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346200 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.358031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346205 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346209 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346213 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346220 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346224 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346228 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346232 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346239 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346245 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346249 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346254 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346258 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346263 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346267 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346271 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346275 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346279 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346284 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346288 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.358526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346292 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346296 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346301 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346305 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346310 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346314 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346323 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346328 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346332 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346338 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346342 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346347 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346351 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346356 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.346397 5894 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346462 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.359000 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346472 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346477 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346482 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346486 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346490 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346495 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346499 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346503 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346508 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346512 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346517 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346521 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346525 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346530 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346534 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346538 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346543 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346547 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346551 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.359391 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346555 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346559 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346563 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346567 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346573 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346577 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346583 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346590 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346594 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346599 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346603 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346607 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346612 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346616 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346621 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346625 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346649 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346655 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346659 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.359944 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346664 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346669 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346673 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346677 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346681 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346686 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346690 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346694 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346702 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346707 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346718 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346723 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346727 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346731 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346736 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346740 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346744 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346749 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346754 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346758 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.360401 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346765 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346769 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346774 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346778 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346784 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346791 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346798 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346803 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346807 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346812 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346816 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346820 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346825 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346829 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346833 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346838 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346842 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346846 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346850 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346855 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.360894 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346859 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346863 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346867 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346872 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346876 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346880 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346885 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346889 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346894 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346898 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346902 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346907 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346916 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346924 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346928 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346933 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346938 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346942 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346946 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.361374 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346950 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346954 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346959 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346964 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346968 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346972 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346976 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.346980 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352038 5894 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352054 5894 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352074 5894 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352079 5894 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352145 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352150 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352153 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352156 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352159 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352162 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352165 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352167 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.361856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352170 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352172 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352175 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352178 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352181 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352183 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352186 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352189 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352192 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352195 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352197 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352200 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352202 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352205 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352208 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352210 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352213 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352216 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352222 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352225 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.362329 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352228 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352231 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352233 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352236 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352239 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352241 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352244 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352246 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352249 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352252 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352254 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352257 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352260 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352262 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352265 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352267 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352270 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352272 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352276 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352279 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.362834 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352282 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352284 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352287 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352290 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352293 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352295 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352298 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352301 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352303 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352306 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352309 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352311 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352314 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352316 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352319 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352322 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352324 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352327 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352329 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.363342 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352332 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352334 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352337 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352340 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352342 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352345 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352347 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352350 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352353 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352356 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352359 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352361 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352366 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352371 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352374 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352377 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352380 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352382 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352385 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.363856 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352389 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352391 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352394 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352397 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352400 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352402 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352405 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352408 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352410 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352413 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352416 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352418 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352421 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352424 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352426 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352429 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352431 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352434 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352438 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.364346 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352441 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352469 5894 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352603 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352609 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352612 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352615 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352618 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352621 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352624 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352628 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352642 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352645 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352648 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352650 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352653 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352656 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.364993 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352659 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352662 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352665 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352667 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352670 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352673 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352676 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352679 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352681 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352684 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352687 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352690 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352693 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352696 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352698 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352701 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352704 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352706 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352709 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.365371 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352711 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352714 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352717 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352720 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352741 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352745 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352749 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352752 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352754 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352757 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352760 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352763 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352765 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352768 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352770 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352773 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352775 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352778 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352781 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352783 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.366031 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352785 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352788 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352791 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352793 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352796 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352798 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352801 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352804 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352807 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352810 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352812 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352816 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352819 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352823 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352826 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352829 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352831 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352834 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352837 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.366526 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352839 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352842 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352845 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352847 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352850 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352853 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352855 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352858 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352862 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352865 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352868 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352871 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352873 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352876 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352878 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352881 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352884 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352886 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352889 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.367022 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352892 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352894 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352897 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352899 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352902 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352904 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352907 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352910 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352912 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352915 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352917 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352920 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352922 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352925 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352928 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.352951 5894 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 06:57:16.367807 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352984 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352988 5894 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352991 5894 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352994 5894 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.352997 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353000 5894 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353002 5894 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353005 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353007 5894 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353010 5894 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353016 5894 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353018 5894 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353021 5894 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353024 5894 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353026 5894 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353029 5894 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353032 5894 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353035 5894 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353038 5894 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353040 5894 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 06:57:16.368465 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353043 5894 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353046 5894 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353048 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353051 5894 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353054 5894 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353056 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353059 5894 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353062 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353064 5894 feature_gate.go:386] unrecognized feature gate: Example Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353067 5894 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353070 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353073 5894 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353076 5894 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353079 5894 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353081 5894 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353083 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353086 5894 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353089 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353091 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353094 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 06:57:16.369039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353097 5894 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353099 5894 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353104 5894 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353106 5894 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353109 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353112 5894 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353114 5894 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353117 5894 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353121 5894 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353124 5894 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353126 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353129 5894 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353132 5894 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353135 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353138 5894 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353140 5894 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353143 5894 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353146 5894 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353148 5894 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353152 5894 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 06:57:16.369550 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353154 5894 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353157 5894 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353159 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353162 5894 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353165 5894 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353168 5894 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353170 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353173 5894 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353176 5894 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353178 5894 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353181 5894 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353183 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353186 5894 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353188 5894 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353192 5894 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353195 5894 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353198 5894 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353200 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353203 5894 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 06:57:16.370039 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353207 5894 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353212 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353216 5894 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353219 5894 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353221 5894 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353224 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353226 5894 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353229 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353232 5894 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353234 5894 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353237 5894 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353240 5894 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353242 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353245 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353247 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353250 5894 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353254 5894 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353258 5894 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353262 5894 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 06:57:16.370513 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353266 5894 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353269 5894 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353272 5894 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353274 5894 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353277 5894 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353280 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353282 5894 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:16.353285 5894 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.353336 5894 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354307 5894 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354363 5894 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354544 5894 server.go:1014] "Starting client certificate rotation" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354655 5894 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354707 5894 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 06:51:04 +0000 UTC" deadline="2028-03-26 06:11:50.983870604 +0000 UTC" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.354757 5894 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12311h14m34.629115964s" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.355014 5894 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:57:16.371006 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.356118 5894 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 06:57:16.371428 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.357511 5894 log.go:25] "Validated CRI v1 runtime API" Oct 30 06:57:16.371428 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.359455 5894 log.go:25] "Validated CRI v1 image API" Oct 30 06:57:16.371428 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.360214 5894 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 06:57:16.371428 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.362555 5894 fs.go:135] Filesystem UUIDs: map[35df0802-4ff4-46f6-9add-e53b2277e98c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 846e5293-d76a-4dff-8954-09f47d5aeeda:/dev/nvme0n1p4] Oct 30 06:57:16.371556 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.362574 5894 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498/userdata/shm major:0 minor:281 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b/userdata/shm major:0 minor:277 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9/userdata/shm major:0 minor:450 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b/userdata/shm major:0 minor:401 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0/userdata/shm major:0 minor:279 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215/userdata/shm major:0 minor:500 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0/userdata/shm major:0 minor:495 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~projected/kube-api-access-brbvw:{mountpoint:/var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~projected/kube-api-access-brbvw major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20e60768-4e85-440c-8ddf-cb8f91fdd289/volumes/kubernetes.io~projected/kube-api-access-fr2vt:{mountpoint:/var/lib/kubelet/pods/20e60768-4e85-440c-8ddf-cb8f91fdd289/volumes/kubernetes.io~projected/kube-api-access-fr2vt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~projected/kube-api-access-fj55v:{mountpoint:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~projected/kube-api-access-fj55v major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~projected/kube-api-access-8gvn9:{mountpoint:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~projected/kube-api-access-8gvn9 major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~projected/kube-api-access-4t8lp:{mountpoint:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~projected/kube-api-access-4t8lp major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5300a292-d959-4d7b-a1fe-b1a1f81efd71/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5300a292-d959-4d7b-a1fe-b1a1f81efd71/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~projected/kube-api-access-rx82n:{mountpoint:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~projected/kube-api-access-rx82n major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~projected/kube-api-access-zbkd2:{mountpoint:/var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~projected/kube-api-access-zbkd2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/679a9341-9428-4ae3-a14c-e72618082e3b/volumes/kubernetes.io~projected/kube-api-access-t9qtb:{mountpoint:/var/lib/kubelet/pods/679a9341-9428-4ae3-a14c-e72618082e3b/volumes/kubernetes.io~projected/kube-api-access-t9qtb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ca9fe1e-35ee-469e-b4af-7c13480101f7/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6ca9fe1e-35ee-469e-b4af-7c13480101f7/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/config-reloader/2 major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/prometheus/15 major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/kube-api-access-fz4fh:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/kube-api-access-fz4fh major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/tls-assets major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/config major:0 minor:439 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/web-config major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~projected/kube-api-access-n69kj:{mountpoint:/var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~projected/kube-api-access-n69kj major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~secret/metrics-certs major:0 minor:492 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~projected/kube-api-access-wd6nw:{mountpoint:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~projected/kube-api-access-wd6nw major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/869e41d6-ed18-44a7-96ec-74efb259f79f/volumes/kubernetes.io~projected/kube-api-access-6w2qb:{mountpoint:/var/lib/kubelet/pods/869e41d6-ed18-44a7-96ec-74efb259f79f/volumes/kubernetes.io~projected/kube-api-access-6w2qb major:0 minor:497 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b2a4773-14f8-467d-a46f-a05b38375900/volumes/kubernetes.io~projected/kube-api-access-72bbc:{mountpoint:/var/lib/kubelet/pods/8b2a4773-14f8-467d-a46f-a05b38375900/volumes/kubernetes.io~projected/kube-api-access-72bbc major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d4538a8-12ff-45c9-ae2c-fcc65413dae1/volumes/kubernetes.io~projected/kube-api-access-8mwv7:{mountpoint:/var/lib/kubelet/pods/8d4538a8-12ff-45c9-ae2c-fcc65413dae1/volumes/kubernetes.io~projected/kube-api-access-8mwv7 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~projected/kube-api-access-dn9sn:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~projected/kube-api-access-dn9sn major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/alertmanager/9 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/config-reloader/6 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~empty-dir/config-out major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/kube-api-access-vbn5z:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/kube-api-access-vbn5z major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/tls-assets major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/config-volume major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/web-config major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e4f6bc7-4192-4165-b16d-df0db49aedd9/volumes/kubernetes.io~projected/kube-api-access-zw9hj:{mountpoint:/var/lib/kubelet/pods/9e4f6bc7-4192-4165-b16d-df0db49aedd9/volumes/kubernetes.io~projected/kube-api-access-zw9hj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~projected/kube-api-access-wtb4k:{mountpoint:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~projected/kube-api-access-wtb4k major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~projected/kube-api-access-fwv82:{mountpoint:/var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~projected/kube-api-access-fwv82 major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~projected/kube-api-access-4wv2r:{mountpoint:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~projected/kube-api-access-4wv2r major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~projected/kube-api-access-wcmk2:{mountpoint:/var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~projected/kube-api-access-wcmk2 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0b0503e-7393-44db-8f5c-1aaaf2043b07/volumes/kubernetes.io~projected/kube-api-access-5jzsb:{mountpoint:/var/lib/kubelet/pods/f0b0503e-7393-44db-8f5c-1aaaf2043b07/volumes/kubernetes.io~projected/kube-api-access-5jzsb major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4c89bc0fe5db602e10adf95f159be262b4bbbafb8cc002e885362ca1213ef363/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4dd439f6c89931b8e62335aae976ddc11d3c92665056db5445e67c00030c8ed9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/16510ebee0d306a8b7a08129e56c7b0b7e02940f389e055806e366be8b7010ea/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/ddc192c680f81bc9a2173bcc621dd4cb039282175102816a4649cb95e448b2db/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2318d00412d6f587e97e3fde6cb94c56d36fd0136aca093859451dbe7748c63f/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d2466f0bf90654ea2951b2bda0001a8e012d37ff6e428b59226cda4be40c64be/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/deb5ce7efbf0fe0ee0167e08376d2216e26340598db871147c1bbe0e976e646a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a39218e194a63e4879dee58dfe8b452d274d190e3b5486fe5709329597675a19/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/37ff4b577abcdc53b9051d4617e3693622e21934547d990ef5817004bff81cd2/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/7b578e2024dbd86e20cb468c2aba4ac806c4254486e1c5e61e04ad06fc807784/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/bae57469dfb7aadcc5fa586598d857e3a5036a544ed7186bd17296b387376a22/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/bb4784e4ae28d6b1ad5314bc83b1e0bf39b1e9641fab7ceae9914ceec27be6d1/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a1b84e173bad8d74b019187a282c5be6c9ec70a3f1ea8ec4a126301800d74e03/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/858d53bf2b3de7e2bd88ffe385bfb0240bfbde1888d0e9dacdf8b0b11293f590/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/4148d60933af4ddccba74ab176855e4d25c26ca442c8172076a90804e17f5bfa/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1049077918f57c697663e0f43a67f36e97423ddc6b985d40e66e2bc1aaa8ad4c/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/1423aacc5fb8a0a2f0e76c181fb35e6f6a0e91d5d2ce72b0241db2804982a297/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e917ebd2bc4344fdfaf7cbc2862888aae06ace75d85a7d693988bd4c894a4d50/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6bae44e3c440bc1121e58ba35f44c09a89575158caa4c5e2ab66f2db9bde6982/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6f485f937e77ad7a27e7b46c391773678dd93e82e287e31829caca192fab199f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3ef00b64f48764afaffe059428d647806b8b7840ce0e9156c99aaf5a75b88db0/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/90cd008792ebe81b04a6bc6a9610ad629f1867aef26a42e72c08bcd427497aa3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8a00cd696a54712ecb3326c464738e02136dc3b909cef49da9ee60b266713613/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/4065c3d1887fb74d156f9b4c2fc653429fb348af3507e924aacfc82a74a14a67/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/dc400d0ea3da592b8802f88ad417579b5ad341b4d083eb4f77ffea34165a4e27/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/0d68bf8fdc075a2960fe71cd77204efa591773660bb5542f26bc1f362dd0a095/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/d765f7dc9b396b5d2c23a54aa8e69fea338c33c666f8f965ac76d7ed701c3c90/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/7916edbe7aebcd7e587543ef53fde398845890c5286e44d841e6370405393c7f/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/b6874548ebb8c229a978fbd5cf3f409c6620c52bdb30c2e2e8b7a31981521da0/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/beeddc8e0e75301f4e9fad57fc4e833dd9de3609ad90fc1528a94c1042f52549/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/d4cba5a5eb14a1f226b70012148e404bc19f5e3ee50b25b8319bbcfa8601a707/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/045a8a1bf265a9671f01bc8ae4edb79213b274d00ba552b914ecd7dd738f11bc/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/ae918535072ef19071f93485c9425106cec4153a79c598f2c92fd9f601b3f416/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/fa81254797809b9bc05c120d3b7293678dced089b2c78cf2bf0fefc871933da5/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/bceba07ca64634c771a51257ab25e1c26efd3cd4fc44070de2ec36ef67f4afe5/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/9928e516f46b879313e9674295ad57e1f9b06fc11eea83de052f6215c91b1896/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/1bb92c6a7e176499121116c2edf3adc0ce43403a3d1854bfdb7bd1cca6ba1012/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/a69f35ce8c8f1a914c60b83626c32f6482541e1ceb469205528839220ad9e357/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6ebc60657f4d8de3b6d79fedd8b372024c4728548a986a2dd59194642d594a43/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/c774dcbdd990aa4080d815a876638abc20aca379be8fe05a5d4bad7c7b9ab2ac/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/1c410479754e7da0841835e7fcaa16f0e08b6d469abf72041fd2d55c1b389315/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/c121231747c9008f5983f37415a18ee01a4ef4c807f0d9ca07e94539b6c1e9b7/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/75a35c93648c10ad749d59155e82b5c2941bdd9112d2a3f968a95ad3af0c5c96/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/136829adc3835aa66444c16d6c5b41f3f82e061a96705592dfbab128aa4737a6/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/342bbcda2fe0f1bd7622a3e13221fc105550fb3c2b73e0610287169be8ae6a4c/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/a4eab93185287722d739024836aaf9251725cf4ef4eaf5f2801992ec1f76b304/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/a6f42055906b3fb4af56391193fdc1c4ff64d7666a45b5bc66d11b31c0d68a96/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/bbadcae3188e10a5bd513cbfcf80350e9e825ebff55b11b18e346d6cf82d1dac/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/fdaede070527d609d2fc78b1263a845781a786488b082aa8d1b46720b5d86c2d/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/e6eb1f643bbd99b690122f2da8b5f5395ea4a8fa23c5b80fea7bee2d15c3cb7c/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/0be87fb30f27fd5931f31bf4c6ac5649d7b46cec350d65d3834e652c725b76ba/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/7056697f4d08260defaf4364a462f34e7695361396117539a789f71bf17a07f4/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/e47ab4a2c3d448910affcfa0c199210dd8d4f88a5219a8a77292d70825b9a88d/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/0f9d6dca49cbdb16cffefeb8cf2fb18f3a37ba5f38f7f932e3170f3bdc9d1f0f/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/fe5ba922780af480c06ebef8cfed949889a480aefe84b07f975930267f94af4b/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/7c514d173020b8d1877668f95233baed778785800bc05b103900c3d522b8f4f5/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/e8d893165e699f7913c679fa1ec45a4b36e8a640b5130f5fa62fb4e514bf6685/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/d9bd25a433e36a6efbe1bbd56269a8515b3159339eeb6543d098e2fa4b44a548/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/729ff7e3f09b3fd728bb139417b597ab3526787da860f22e8df8cd0762305d60/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/b0bf3b1f889b4663ebef632517416dce8bbae0e1d01e151a21d66700bd81a8ea/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/fa5d7af0e4c3a835f615463499a203805eb10163eaceeb1295cc880090ed17a3/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/db9a5f569d08dcfa7a1155fe7be1c0610a8e678aa88400ff292e8370da9887c8/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/e0892d099099e9c48102bef82aa7a50c6cedc4e4707886046a74d52f1be66504/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/0734dc68de3c1fa903c7b5cc8d2dee093ccc8ea6ba3d1badaf72328f4048c134/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/e45efc21463e250e1dcaf17ec3b664b32a1853e6c5f8f23f32d9daa6cbd3ec51/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/65d50b03be21c1f23fc017d6d23c62f34b0bf64ec087a789f9c7951881841f12/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/b6be017a296d9f577d8e55d36f0c709042765d6ed04834697491e162fb443e86/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/5e39eb1ecea7eb2ed13605f2f5f0b01ec61a595668731034496170a500d78a7a/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/79d50825379cf876253b7f99b41cd0dea61d730fefafae3943d05f3147717d98/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/afc69c60b9ab9b239b2cd9e667364c889bd681cc42ef728afc4b81ad50ab67c8/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-502:{mountpoint:/var/lib/containers/storage/overlay/4f2156fa28f4594dc67533a1f52d2c483d41c89c35d72c86feb70f5acdf183d6/merged major:0 minor:502 fsType:overlay blockSize:0} overlay_0-504:{mountpoint:/var/lib/containers/storage/overlay/dfc9421485579f33eacf876d3897b667f4ce2f14ce02e7450bec9d958144c30f/merged major:0 minor:504 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/cafd88ebf6c028a0e05e61e789bdd1ad73bc84cf625dcfed47c2c81ad02b0035/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/490eac28e5b93d309020f0d88a152733922554d90b2f25445e487752769c911d/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/838834e75e3812550d8f1fe2cb9c6c52cbdaba7a101e82645d47c589b8d77deb/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/6207013adcfed72aa0c22f89ec3673229b36321faf819ba3c48a415dd68c1a0b/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-519:{mountpoint:/var/lib/containers/storage/overlay/c5732b3d6460d1ff40e1c810895fd39ba38af8c6dcdcefa9f05ae66d86fda115/merged major:0 minor:519 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/856fa3746ff95fff833b04e8f9796eeaa7134d50f26ec304128e37aeea7d7a5d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/04f48e4b9acb210fd0ea9a9e37cc561e1cb88ab5dd0c2f781b5541d14da46706/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/bbe2b48f368f755a09e27800481d625cb76ae844004f6810d918500ad11eaca8/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/91acae29c027bf425b41e9471570c994a8d26cb654e801fd220ded449521da05/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/e1d4e326ae7b750e4b909bbd2092a4097621e9535e7c633a6c1b2669ea935f84/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/5b7fd94ea98f25dd2772694d260d09ccd5463989917626d4b8bc3f21fc1784e5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f717f312bc45b1101d3aaa1ef7248c5fdc8ccb347a0f7f77cce45ed7a04c517a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a4f2eb63e901efe2cf7a34dc0256a430be1fe67701d876120bbaa4faa35884f1/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/9cdfbf8a05bd2a30e06501c68abd23434f1ff09f36ec123567aa2b066128482f/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 06:57:16.372548 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.372086 5894 manager.go:217] Machine: {Timestamp:2026-10-30 06:57:16.371601571 +0000 UTC m=+0.062518999 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3094593 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b8761dc9b45bed08e3a82ed415a61 SystemUUID:ec2b8761-dc9b-45be-d08e-3a82ed415a61 BootID:d215f6d4-631e-4431-8bb5-92039bf0d711 Filesystems:[{Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:365 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~projected/kube-api-access-fwv82 DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~projected/kube-api-access-4t8lp DeviceMajor:0 DeviceMinor:275 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:492 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d4538a8-12ff-45c9-ae2c-fcc65413dae1/volumes/kubernetes.io~projected/kube-api-access-8mwv7 DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~projected/kube-api-access-brbvw DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:302 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:443 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~projected/kube-api-access-zbkd2 DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68cdfaa618ded2c1b2747eb2c0cb6914151e259c58c842c098b51802af9b0020/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-502 DeviceMajor:0 DeviceMinor:502 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6ca9fe1e-35ee-469e-b4af-7c13480101f7/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:488 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8e6da5523543f6326510172b27f1dad0954de32bd2660ae311a61d65b1e1f5b/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:301 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~projected/kube-api-access-wtb4k DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ded0db88bb55a832729df9f4e70481c306d8b1a8ff308bd210f46dfddcba847a/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:303 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:441 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:447 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/679a9341-9428-4ae3-a14c-e72618082e3b/volumes/kubernetes.io~projected/kube-api-access-t9qtb DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4ba9488a84821a3f774e148da93e534032e094698def6903e7d9d37f0df6a0a/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~projected/kube-api-access-4wv2r DeviceMajor:0 DeviceMinor:276 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6da7f6388159b513cf26ac9642c14359f8242f7fa45e15a9750dc16eb99e6de/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:448 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcc3458c8bcb277ece1fa4467169924e5364151cbd8e4bddadf6b738023ebdf0/userdata/shm DeviceMajor:0 DeviceMinor:495 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b101f390b2642c340143c69b55460fad25609c3049c9967cef3df9a0a62ffd1d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:350 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6215b7c5-298f-481a-b3ce-884ad7382b26/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~projected/kube-api-access-wcmk2 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:297 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:434 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:439 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5300a292-d959-4d7b-a1fe-b1a1f81efd71/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77bfd548-5933-4191-a2b2-a9ec19692a83/volumes/kubernetes.io~projected/kube-api-access-n69kj DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9/userdata/shm DeviceMajor:0 DeviceMinor:450 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-504 DeviceMajor:0 DeviceMinor:504 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f29819d3f9ab2822676ef6095f2ed9420765217bac8e43122dfaf0f9585f9e0/userdata/shm DeviceMajor:0 DeviceMinor:279 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3122922f-c624-4694-9e9d-17893f6d9af8/volumes/kubernetes.io~projected/kube-api-access-fj55v DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20e60768-4e85-440c-8ddf-cb8f91fdd289/volumes/kubernetes.io~projected/kube-api-access-fr2vt DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78fcc7de168d81c2801b9d2fbf1d1ca678b797261d34785e49004411d208af03/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01a74e65-a10a-423a-b889-8105cdb97b1b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/799484a3-d828-4795-b681-710baad6a32c/volumes/kubernetes.io~projected/kube-api-access-wd6nw DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b2a4773-14f8-467d-a46f-a05b38375900/volumes/kubernetes.io~projected/kube-api-access-72bbc DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b1429333-ac51-49ee-8173-806b8ae7aeaf/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:381 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/25c7bde42424636be0b89f6f93952b14cf364639fb45ce3c5bcba57062ac6f8b/userdata/shm DeviceMajor:0 DeviceMinor:277 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:307 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~projected/kube-api-access-vbn5z DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~projected/kube-api-access-dn9sn DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:440 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/kube-api-access-fz4fh DeviceMajor:0 DeviceMinor:449 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3355f7009f41f76fdd6b2632d8ea3a93da5cc88de3faac97bd7c181a7f276072/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/869e41d6-ed18-44a7-96ec-74efb259f79f/volumes/kubernetes.io~projected/kube-api-access-6w2qb DeviceMajor:0 DeviceMinor:497 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e4f6bc7-4192-4165-b16d-df0db49aedd9/volumes/kubernetes.io~projected/kube-api-access-zw9hj DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~projected/kube-api-access-8gvn9 DeviceMajor:0 DeviceMinor:274 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:364 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:442 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5f0bec43bfcf0a58c1d3ba6a36e54cf7e56e21578464dd1174b9f45d575f215/userdata/shm DeviceMajor:0 DeviceMinor:500 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ac5ca8962ede14c3d62f457403d9d7ce8ed6b4f87d26a43c2e71e701157156e/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fb72182d2099aa251256edb1b18fb690baef60ce090dc694c207ff0781f809a/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8189a712e9817870b2a6653b4ab03bd2481a93637f5c12399c1b32eda2336c7b/userdata/shm DeviceMajor:0 DeviceMinor:401 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a87bd4fc-9395-46bf-87eb-13a159db993a/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:304 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:269 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~projected/kube-api-access-rx82n DeviceMajor:0 DeviceMinor:367 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:446 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a7f4421-c460-4690-b755-a6fef7f2e487/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:308 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:444 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:270 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/99c07bc5-c5be-4d09-9f96-1e50f37423be/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-519 DeviceMajor:0 DeviceMinor:519 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fba918a2af550ecfdc7b893da0136c17bea4f787d28d0670335d825f72abc383/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e53843be29448afe991926bb658f6b51ba742fabfdf6efe82ef562a871e0c87/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:445 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3fc44517-a592-4c5e-9a42-96047cabe380/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:264 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:273 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f0b0503e-7393-44db-8f5c-1aaaf2043b07/volumes/kubernetes.io~projected/kube-api-access-5jzsb DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/697422f4db5b459637426901523e922d58af4978b5206e40011be8bb5f3bf652/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dcc831f3187d11e5e0dcbffbe167f508a1969f649ea63360425fd26999c8dd3/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/128725bdd1ca2170305ea92065c174b4e5b6269a05eac3317c2c58651cf695af/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f080f672-8cc7-4b00-a975-14ea5c5b1ae8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b656ad4bcdffdacbc0d517a9f0fd7f7fc9464c32c3e0197da9295f3cd881498/userdata/shm DeviceMajor:0 DeviceMinor:281 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:128725bdd1ca217 MacAddress:96:91:68:e3:84:e5 Speed:10000 Mtu:8901} {Name:25c7bde42424636 MacAddress:be:6f:6a:8b:2d:10 Speed:10000 Mtu:8901} {Name:2d00763a83420b6 MacAddress:d6:3d:59:e4:8e:48 Speed:10000 Mtu:8901} {Name:2fb72182d2099aa MacAddress:aa:87:09:7f:86:7b Speed:10000 Mtu:8901} {Name:3355f7009f41f76 MacAddress:9e:f0:e1:07:2d:f0 Speed:10000 Mtu:8901} {Name:68cdfaa618ded2c MacAddress:d2:a8:1a:f0:de:7e Speed:10000 Mtu:8901} {Name:78fcc7de168d81c MacAddress:ce:40:e9:40:29:f1 Speed:10000 Mtu:8901} {Name:7e53843be29448a MacAddress:1e:13:11:e3:c3:82 Speed:10000 Mtu:8901} {Name:8189a712e981787 MacAddress:3e:cf:bd:0b:e0:ed Speed:10000 Mtu:8901} {Name:9f29819d3f9ab28 MacAddress:16:ec:f3:86:02:1e Speed:10000 Mtu:8901} {Name:b4ba9488a84821a MacAddress:32:43:3f:77:de:c8 Speed:10000 Mtu:8901} {Name:b5f0bec43bfcf0a MacAddress:e2:8f:8d:cb:18:2a Speed:10000 Mtu:8901} {Name:bf4628c04defbdc MacAddress:06:41:25:fa:19:b6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:a7:64:5b Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:68:06:60:d6:7c Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:a7:64:5b Speed:0 Mtu:9001} {Name:fcc3458c8bcb277 MacAddress:a2:77:8b:dd:76:07 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:02:26:1f:68:3d:d4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7e:61:d8:64:6d:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 06:57:16.372936 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.372553 5894 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 06:57:16.372936 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.372601 5894 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 06:57:16.373000 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.372982 5894 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 06:57:16.373124 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373002 5894 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-90.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 06:57:16.373165 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373126 5894 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 06:57:16.373165 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373133 5894 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 06:57:16.373165 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373144 5894 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:57:16.373165 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373163 5894 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 06:57:16.373271 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373249 5894 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 06:57:16.373444 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373434 5894 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 06:57:16.373560 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373551 5894 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 06:57:16.373600 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373591 5894 kubelet.go:493] "Attempting to sync node with API server" Oct 30 06:57:16.373653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373605 5894 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 06:57:16.373653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373616 5894 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 06:57:16.373653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373624 5894 kubelet.go:405] "Adding apiserver pod source" Oct 30 06:57:16.373653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.373649 5894 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 06:57:16.374722 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.374699 5894 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:57:16.374722 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.374724 5894 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:57:16.375239 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.375219 5894 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 06:57:16.378454 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378437 5894 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 06:57:16.378454 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378452 5894 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 06:57:16.378573 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378548 5894 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 06:57:16.378780 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378767 5894 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 06:57:16.378813 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378798 5894 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 06:57:16.378918 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378908 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 06:57:16.378947 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378928 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 06:57:16.378947 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378938 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 06:57:16.378947 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378945 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378950 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378956 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378963 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378968 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378975 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378980 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378988 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.378998 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.379025 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 06:57:16.379028 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.379030 5894 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 06:57:16.382471 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.382457 5894 server.go:1263] "Started kubelet" Oct 30 06:57:16.383590 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.382607 5894 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 06:57:16.383590 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.382841 5894 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 06:57:16.383590 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.383181 5894 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 06:57:16.383590 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.383507 5894 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 06:57:16.383103 ip-10-0-143-90 systemd[1]: Started Kubernetes Kubelet. Oct 30 06:57:16.385989 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.385108 5894 server.go:317] "Adding debug handlers to kubelet server" Oct 30 06:57:16.389864 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.389282 5894 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 06:57:16.389864 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.389389 5894 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 06:57:16.389864 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.389407 5894 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 06:57:16.389864 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.389517 5894 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 06:57:16.389864 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.389373 5894 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-90.ec2.internal\" not found" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392706 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392747 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392762 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392776 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20e60768-4e85-440c-8ddf-cb8f91fdd289" volumeName="kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392788 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3122922f-c624-4694-9e9d-17893f6d9af8" volumeName="kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392799 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392812 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392824 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392839 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392850 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bfd548-5933-4191-a2b2-a9ec19692a83" volumeName="kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392862 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="869e41d6-ed18-44a7-96ec-74efb259f79f" volumeName="kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392875 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5300a292-d959-4d7b-a1fe-b1a1f81efd71" volumeName="kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392886 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b2a4773-14f8-467d-a46f-a05b38375900" volumeName="kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392904 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4538a8-12ff-45c9-ae2c-fcc65413dae1" volumeName="kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392920 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392931 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392942 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392953 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392964 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4f6bc7-4192-4165-b16d-df0db49aedd9" volumeName="kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392976 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20e60768-4e85-440c-8ddf-cb8f91fdd289" volumeName="kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392987 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.392998 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393009 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393020 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799484a3-d828-4795-b681-710baad6a32c" volumeName="kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393032 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799484a3-d828-4795-b681-710baad6a32c" volumeName="kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393043 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4538a8-12ff-45c9-ae2c-fcc65413dae1" volumeName="kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393054 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393067 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393081 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="679a9341-9428-4ae3-a14c-e72618082e3b" volumeName="kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393095 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="679a9341-9428-4ae3-a14c-e72618082e3b" volumeName="kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393106 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393118 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bfd548-5933-4191-a2b2-a9ec19692a83" volumeName="kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393130 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393141 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393152 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a87bd4fc-9395-46bf-87eb-13a159db993a" volumeName="kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393163 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393177 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393188 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393200 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799484a3-d828-4795-b681-710baad6a32c" volumeName="kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393210 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393221 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393233 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a87bd4fc-9395-46bf-87eb-13a159db993a" volumeName="kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393245 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe05ba0-4fc7-4df3-bf3b-af4feadb519d" volumeName="kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393256 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01a74e65-a10a-423a-b889-8105cdb97b1b" volumeName="kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393271 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393283 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="799484a3-d828-4795-b681-710baad6a32c" volumeName="kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393294 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393306 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1429333-ac51-49ee-8173-806b8ae7aeaf" volumeName="kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393317 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe05ba0-4fc7-4df3-bf3b-af4feadb519d" volumeName="kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393329 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0b0503e-7393-44db-8f5c-1aaaf2043b07" volumeName="kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393340 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6215b7c5-298f-481a-b3ce-884ad7382b26" volumeName="kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393351 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393362 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a87bd4fc-9395-46bf-87eb-13a159db993a" volumeName="kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393373 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe05ba0-4fc7-4df3-bf3b-af4feadb519d" volumeName="kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393385 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f080f672-8cc7-4b00-a975-14ea5c5b1ae8" volumeName="kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393396 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f080f672-8cc7-4b00-a975-14ea5c5b1ae8" volumeName="kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393415 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393426 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6215b7c5-298f-481a-b3ce-884ad7382b26" volumeName="kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393437 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ca9fe1e-35ee-469e-b4af-7c13480101f7" volumeName="kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393448 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393461 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393476 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b2a4773-14f8-467d-a46f-a05b38375900" volumeName="kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393488 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393499 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393511 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393523 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393535 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393547 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393558 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4f6bc7-4192-4165-b16d-df0db49aedd9" volumeName="kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393568 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3122922f-c624-4694-9e9d-17893f6d9af8" volumeName="kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393580 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393590 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6215b7c5-298f-481a-b3ce-884ad7382b26" volumeName="kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393601 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393612 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f080f672-8cc7-4b00-a975-14ea5c5b1ae8" volumeName="kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393622 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393650 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="679a9341-9428-4ae3-a14c-e72618082e3b" volumeName="kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393662 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4538a8-12ff-45c9-ae2c-fcc65413dae1" volumeName="kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393674 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a87bd4fc-9395-46bf-87eb-13a159db993a" volumeName="kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393685 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1429333-ac51-49ee-8173-806b8ae7aeaf" volumeName="kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393695 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393706 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393716 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393726 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393736 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393748 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393758 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5300a292-d959-4d7b-a1fe-b1a1f81efd71" volumeName="kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393768 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393780 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393790 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393800 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393812 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393822 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="679a9341-9428-4ae3-a14c-e72618082e3b" volumeName="kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393835 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6daadbca-3412-44d6-85ab-94e80b0ca09a" volumeName="kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393849 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393860 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393871 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a87bd4fc-9395-46bf-87eb-13a159db993a" volumeName="kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393883 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbe05ba0-4fc7-4df3-bf3b-af4feadb519d" volumeName="kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393916 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" volumeName="kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393929 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393940 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393951 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a7f4421-c460-4690-b755-a6fef7f2e487" volumeName="kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393962 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99c07bc5-c5be-4d09-9f96-1e50f37423be" volumeName="kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393975 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7d331-9233-4871-9d2f-6961c18ca460" volumeName="kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393986 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f080f672-8cc7-4b00-a975-14ea5c5b1ae8" volumeName="kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.393996 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01a74e65-a10a-423a-b889-8105cdb97b1b" volumeName="kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394008 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3122922f-c624-4694-9e9d-17893f6d9af8" volumeName="kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394019 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394031 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fc44517-a592-4c5e-9a42-96047cabe380" volumeName="kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394043 5894 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6215b7c5-298f-481a-b3ce-884ad7382b26" volumeName="kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls" seLinuxMountContext="" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394054 5894 reconstruct.go:97] "Volume reconstruction finished" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394061 5894 reconciler.go:29] "Reconciler: start to sync state" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.389298 5894 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394225 5894 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 06:51:06 +0000 UTC" deadline="2028-08-02 04:39:34.6863357 +0000 UTC" Oct 30 06:57:16.394345 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394257 5894 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15405h42m18.292081174s" Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394661 5894 factory.go:55] Registering systemd factory Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.394681 5894 factory.go:223] Registration of the systemd container factory successfully Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396076 5894 factory.go:153] Registering CRI-O factory Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396095 5894 factory.go:223] Registration of the crio container factory successfully Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396153 5894 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396181 5894 factory.go:103] Registering Raw factory Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396198 5894 manager.go:1199] Started watching for new ooms in manager Oct 30 06:57:16.399007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.396618 5894 manager.go:319] Starting recovery of all containers Oct 30 06:57:16.421334 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.421301 5894 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 06:57:16.423060 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.423033 5894 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 06:57:16.423191 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.423071 5894 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 06:57:16.423191 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.423096 5894 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 06:57:16.423191 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.423143 5894 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 06:57:16.440977 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.440931 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e" exitCode=0 Oct 30 06:57:16.447272 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.447244 5894 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:57:16.447433 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.447417 5894 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 06:57:16.447493 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.447453 5894 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:57:16.449131 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.449106 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028" exitCode=0 Oct 30 06:57:16.457505 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.457479 5894 generic.go:356] "Generic (PLEG): container finished" podID="e2aec6afc092b0a7711821d66a44d1c3" containerID="afc147c4329cbdff2bdd783d9a3ec4e45ac9e01edf5327268b6ebb8489d787fd" exitCode=0 Oct 30 06:57:16.462189 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.462157 5894 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:57:16.468307 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.468277 5894 generic.go:356] "Generic (PLEG): container finished" podID="3ec28222-b8cd-4a03-8a22-3a0d81e5b50f" containerID="1b5fba996e3ab1818ab3a2320609ac4fe3d690b8cf6e52ec90e27ac2dbd927b7" exitCode=0 Oct 30 06:57:16.475273 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.475254 5894 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 06:57:16.487410 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487387 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="f88bd37620a2d217713c85abbb87a027b78af639b0cbe288177f4e8827b93c44" exitCode=0 Oct 30 06:57:16.487410 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487410 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="d3acbfdbb611c5e8109c554d3ad4ab3f735e4be6165908c24928c51daf241725" exitCode=0 Oct 30 06:57:16.487546 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487417 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="023d64566d1c9c9bea5b4959380648a2886fb06a2bc919b9f5a216925512b134" exitCode=0 Oct 30 06:57:16.487546 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487423 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="2809d178eb1f93c6431400a907bfdfe73fff9a187e491c0c60b3650d1662f751" exitCode=0 Oct 30 06:57:16.487546 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487428 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="5023f8531c16e2f37efb8bf04fa4c3533a426178051efc8f5b2874bc8a0389ed" exitCode=0 Oct 30 06:57:16.487546 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.487433 5894 generic.go:356] "Generic (PLEG): container finished" podID="679a9341-9428-4ae3-a14c-e72618082e3b" containerID="70d62cb267998c80d168a6539c9ba4ef9b27f891043657da5c3522256efd31c3" exitCode=0 Oct 30 06:57:16.523282 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.523225 5894 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 06:57:16.525385 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.525367 5894 manager.go:324] Recovery completed Oct 30 06:57:16.550926 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.550897 5894 cpu_manager.go:226] "Starting" policy="none" Oct 30 06:57:16.550926 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.550918 5894 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 06:57:16.551093 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.550938 5894 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 06:57:16.551093 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551069 5894 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 06:57:16.551093 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551080 5894 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 06:57:16.551093 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551096 5894 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 06:57:16.551249 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551102 5894 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 06:57:16.551249 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551107 5894 policy_none.go:50] "Start" Oct 30 06:57:16.551249 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551118 5894 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 06:57:16.551249 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551130 5894 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 06:57:16.551349 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551254 5894 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 06:57:16.551349 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551261 5894 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 06:57:16.551349 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.551267 5894 policy_none.go:44] "Start" Oct 30 06:57:16.555960 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.555933 5894 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 06:57:16.560702 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.560686 5894 manager.go:341] "Starting Device Plugin manager" Oct 30 06:57:16.561283 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.560714 5894 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 06:57:16.561324 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561288 5894 server.go:87] "Starting device plugin registration server" Oct 30 06:57:16.561554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561543 5894 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 06:57:16.561585 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561557 5894 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 06:57:16.561703 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561684 5894 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 06:57:16.561825 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561791 5894 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 06:57:16.561891 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561834 5894 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 06:57:16.561891 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561841 5894 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 06:57:16.561968 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.561892 5894 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T06:57:16.561823715Z","UUID":"aab6a697-b8c8-4437-9e90-f06e9983f9cf","Handler":null,"Name":"","Endpoint":""} Oct 30 06:57:16.563329 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.563311 5894 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 06:57:16.563419 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.563348 5894 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 06:57:16.563593 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.563561 5894 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 06:57:16.661879 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.661849 5894 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 06:57:16.663980 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.663960 5894 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientMemory" Oct 30 06:57:16.664078 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.663990 5894 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 06:57:16.664078 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.664000 5894 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-90.ec2.internal" event="NodeHasSufficientPID" Oct 30 06:57:16.664078 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.664045 5894 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.702619 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.702596 5894 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.702734 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.702677 5894 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.724138 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724103 5894 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal"] Oct 30 06:57:16.724742 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724712 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerStarted","Data":"52148b2c8f8f59bd2bf481511b77232bf648ba230cdacf6111fdcebb758b62c8"} Oct 30 06:57:16.724785 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724753 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerDied","Data":"afc147c4329cbdff2bdd783d9a3ec4e45ac9e01edf5327268b6ebb8489d787fd"} Oct 30 06:57:16.724785 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724774 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" event={"ID":"e2aec6afc092b0a7711821d66a44d1c3","Type":"ContainerStarted","Data":"b368a569c83ab0eae4ab4468068ab1b9ef98f87c9f3d861739534cdc5e7c04b6"} Oct 30 06:57:16.724871 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724812 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" event={"ID":"1648b9f3a1682e8c87ccb70263aa6652","Type":"ContainerStarted","Data":"d7f14e2b8621412b53022697004eb94b1d9021b28ac7b7eef1cf56e76335bc79"} Oct 30 06:57:16.724871 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724733 5894 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.724871 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724829 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" event={"ID":"1648b9f3a1682e8c87ccb70263aa6652","Type":"ContainerStarted","Data":"a2f8eb530ed614ae4ca1351e6d1c2cab5c813f12e7e4e0e70b103649c464ace5"} Oct 30 06:57:16.724996 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.724968 5894 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.750526 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.750501 5894 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:57:16.750657 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.750546 5894 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.750961 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.750948 5894 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:57:16.751025 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:16.750988 5894 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.849406 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.849374 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.849541 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.849413 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.849541 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.849440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.949983 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.949948 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.949983 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.949981 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.950139 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.950002 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.950139 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.950050 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.950139 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.950058 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2aec6afc092b0a7711821d66a44d1c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" (UID: \"e2aec6afc092b0a7711821d66a44d1c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:16.950139 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:16.950118 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1648b9f3a1682e8c87ccb70263aa6652-config\") pod \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" (UID: \"1648b9f3a1682e8c87ccb70263aa6652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:17.374771 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.374738 5894 apiserver.go:52] "Watching apiserver" Oct 30 06:57:17.436598 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.436559 5894 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 06:57:17.437474 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.437449 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sm2hk","openshift-monitoring/thanos-querier-58f5c47444-cfzwr","openshift-network-operator/iptables-alerter-94b4f","openshift-dns/node-resolver-wdkkw","openshift-monitoring/prometheus-k8s-0","openshift-multus/multus-additional-cni-plugins-kwcnj","openshift-dns/dns-default-55nz9","openshift-multus/network-metrics-daemon-xgzmb","openshift-ovn-kubernetes/ovnkube-node-wrvj8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wzqxw","openshift-insights/insights-runtime-extractor-mwst9","openshift-network-diagnostics/network-check-target-7lhl4","openshift-monitoring/metrics-server-65bd9d49b8-7bvx4","kube-system/global-pull-secret-syncer-5n2mb","openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg","openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs","openshift-multus/multus-ff84l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-rhhvm","openshift-monitoring/node-exporter-82xqd","kube-system/konnectivity-agent-4js78","kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal","openshift-monitoring/prometheus-operator-9cd89b74c-qd27n","openshift-monitoring/monitoring-plugin-77f4f76594-65fv7"] Oct 30 06:57:17.455858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.455814 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.455858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.455815 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.455858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.455852 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 06:57:17.456179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.456030 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fp7lr\"" Oct 30 06:57:17.457009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.456979 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 06:57:17.457139 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457028 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.457227 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457152 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zl222\"" Oct 30 06:57:17.457227 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457179 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 06:57:17.457227 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457199 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7f8th\"" Oct 30 06:57:17.457363 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457343 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.457599 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457585 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.457777 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.457763 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r8khf\"" Oct 30 06:57:17.469735 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.469715 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 06:57:17.470137 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.470123 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 06:57:17.470669 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.470630 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.472726 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.472707 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gbbvr\"" Oct 30 06:57:17.472832 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.472808 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 06:57:17.472887 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.472866 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 06:57:17.473056 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.472995 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 06:57:17.473056 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473048 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 06:57:17.473168 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473094 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 06:57:17.473227 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473211 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.473271 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473213 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 06:57:17.473271 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473234 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-s89sh\"" Oct 30 06:57:17.473271 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473214 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.473271 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473240 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.473522 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473497 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 06:57:17.473522 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473508 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.473662 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473619 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.473832 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473817 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 06:57:17.473957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473943 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.474066 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.473960 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 06:57:17.474066 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474043 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 06:57:17.474154 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474075 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 06:57:17.474154 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474092 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.474246 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474211 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 06:57:17.474300 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474286 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z2tvh\"" Oct 30 06:57:17.474354 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474311 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 06:57:17.474354 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474324 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 06:57:17.474354 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474342 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 06:57:17.474475 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474377 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 06:57:17.474475 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474429 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 06:57:17.474475 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474445 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 06:57:17.474599 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474544 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 06:57:17.474599 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474594 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.474719 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474606 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 06:57:17.474719 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474667 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 06:57:17.474804 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474758 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4sbng\"" Oct 30 06:57:17.474851 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474802 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 06:57:17.474851 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474808 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 06:57:17.474851 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474821 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7vgdh\"" Oct 30 06:57:17.474851 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474822 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.475036 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.474853 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 06:57:17.475036 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475001 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cmb4r\"" Oct 30 06:57:17.475129 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475053 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.475129 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475063 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.475129 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475099 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 06:57:17.475129 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475118 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.475129 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475121 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.475588 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475566 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 06:57:17.475779 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.475762 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476221 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-ds4sf\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476243 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476351 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476530 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476557 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-25rrt\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476647 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476628 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7mg7c\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476714 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-mzhb7\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476814 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476908 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.476919 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477093 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477237 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477329 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s5mng\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477370 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477470 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hwx8p\"" Oct 30 06:57:17.477651 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477515 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pnvp5\"" Oct 30 06:57:17.478472 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.477686 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 06:57:17.485120 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485097 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvc75\"" Oct 30 06:57:17.485259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485246 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 06:57:17.485392 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485360 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 06:57:17.485392 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485379 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 06:57:17.485392 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485390 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485387 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485431 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-kkxlz\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485436 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4hfgh\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485459 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-66t9c9j0f6gnq\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485461 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r9nbg\"" Oct 30 06:57:17.485534 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.485437 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6pfvt\"" Oct 30 06:57:17.487186 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487164 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lknhq\"" Oct 30 06:57:17.487259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487202 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cnr7r\"" Oct 30 06:57:17.487259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487173 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-1jff2kns3ngd6\"" Oct 30 06:57:17.487259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487211 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 06:57:17.487259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487186 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 06:57:17.487259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487199 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 06:57:17.487423 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487272 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 06:57:17.487885 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487860 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 06:57:17.487971 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.487893 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 06:57:17.488547 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.488533 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 06:57:17.488606 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.488570 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9kbpgghqali20\"" Oct 30 06:57:17.489466 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489446 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 06:57:17.489538 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489521 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 06:57:17.489614 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489598 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 06:57:17.489695 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489671 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 06:57:17.489778 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489720 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 06:57:17.489778 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489722 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 06:57:17.489884 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.489797 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 06:57:17.490177 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.490162 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 06:57:17.490255 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.490194 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 06:57:17.490699 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.490681 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 06:57:17.490905 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.490890 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 06:57:17.491552 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.491535 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 06:57:17.492796 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.492782 5894 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 06:57:17.496466 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.496450 5894 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:17.496575 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.496563 5894 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:17.523136 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.523109 5894 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:57:17.523254 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:17.523156 5894 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-90.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-90.ec2.internal" Oct 30 06:57:17.523369 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.523354 5894 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 06:57:17.523406 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:17.523396 5894 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-90.ec2.internal" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557277 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557320 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557358 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557382 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557401 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557430 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd6nw\" (UniqueName: \"kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557455 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557482 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557509 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557532 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557549 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9qtb\" (UniqueName: \"kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557582 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557605 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557625 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557660 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557683 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.557701 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557705 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557771 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557809 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557840 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557868 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.557873 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558013 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558046 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wv2r\" (UniqueName: \"kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558074 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558103 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558105 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-daemon-config\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558113 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558136 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558141 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558153 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558166 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558190 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ca9fe1e-35ee-469e-b4af-7c13480101f7-original-pull-secret\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558219 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/3fc44517-a592-4c5e-9a42-96047cabe380-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558239 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mwv7\" (UniqueName: \"kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558262 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558281 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558298 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558313 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558318 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558329 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558344 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558409 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558511 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558534 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558536 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6215b7c5-298f-481a-b3ce-884ad7382b26-tmp-dir\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558551 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558566 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558582 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558602 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558618 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558659 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558676 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558704 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558709 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6215b7c5-298f-481a-b3ce-884ad7382b26-config-volume\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558848 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558872 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558900 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-client-certs\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558925 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558959 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj55v\" (UniqueName: \"kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.558993 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559018 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559047 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559074 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559207 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559236 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cni-binary-copy\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559248 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559277 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559306 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559354 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559361 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559382 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559405 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-script-lib\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559409 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559456 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559489 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559497 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/20e60768-4e85-440c-8ddf-cb8f91fdd289-serviceca\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559523 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559522 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-metrics-client-ca\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559712 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559857 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559858 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559901 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559904 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559943 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559968 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.559990 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560019 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560060 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560091 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560097 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560125 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560146 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560159 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560191 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560221 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560221 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e4f6bc7-4192-4165-b16d-df0db49aedd9-tmp-dir\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560255 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5300a292-d959-4d7b-a1fe-b1a1f81efd71-konnectivity-ca\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:57:17.560337 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560370 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560403 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560423 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77bfd548-5933-4191-a2b2-a9ec19692a83-metrics-certs\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560442 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560471 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bbc\" (UniqueName: \"kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560499 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbvw\" (UniqueName: \"kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560529 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560555 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6215b7c5-298f-481a-b3ce-884ad7382b26-metrics-tls\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560557 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560600 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-tls\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560620 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560624 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-etc-tuned\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560602 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560669 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b1429333-ac51-49ee-8173-806b8ae7aeaf-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560693 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99c07bc5-c5be-4d09-9f96-1e50f37423be-tmp\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560773 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-env-overrides\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560822 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560827 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560853 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560864 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-textfile\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560857 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560966 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.560982 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561027 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561062 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561061 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561077 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561198 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561238 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561246 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a87bd4fc-9395-46bf-87eb-13a159db993a-ovnkube-config\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561355 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561390 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561407 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561461 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561496 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561539 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561587 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01a74e65-a10a-423a-b889-8105cdb97b1b-cert\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561615 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561676 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561627 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561700 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-data-volume\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561711 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561730 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561766 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561794 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561818 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561890 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561936 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.561940 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562043 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn9sn\" (UniqueName: \"kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562065 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5300a292-d959-4d7b-a1fe-b1a1f81efd71-agent-certs\") pod \"konnectivity-agent-4js78\" (UID: \"5300a292-d959-4d7b-a1fe-b1a1f81efd71\") " pod="kube-system/konnectivity-agent-4js78" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562090 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gvn9\" (UniqueName: \"kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562120 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562192 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbkd2\" (UniqueName: \"kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562301 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwv82\" (UniqueName: \"kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562374 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562430 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562489 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562552 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.562557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562586 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69kj\" (UniqueName: \"kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562655 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562693 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562737 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562855 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562917 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzsb\" (UniqueName: \"kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.562922 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563030 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2vt\" (UniqueName: \"kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563128 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563197 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563220 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563257 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563287 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563312 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563340 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563366 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563409 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563446 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563451 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563463 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563481 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563510 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/799484a3-d828-4795-b681-710baad6a32c-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563582 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-secret-metrics-server-tls\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563555 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t8lp\" (UniqueName: \"kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563627 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563676 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a87bd4fc-9395-46bf-87eb-13a159db993a-ovn-node-metrics-cert\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563693 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563739 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/3fc44517-a592-4c5e-9a42-96047cabe380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563747 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx82n\" (UniqueName: \"kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563783 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563813 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563841 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563870 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563919 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563953 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563977 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564023 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.563982 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564059 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564072 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564076 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564110 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564118 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3122922f-c624-4694-9e9d-17893f6d9af8-tmp\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564160 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564190 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564218 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564243 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564344 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564359 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564388 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564407 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564445 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564446 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564497 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-grpc-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564537 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564561 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-accelerators-collector-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564583 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmk2\" (UniqueName: \"kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564657 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564679 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564697 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564734 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564804 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564832 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564859 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtb4k\" (UniqueName: \"kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564903 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564909 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564936 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564968 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564984 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.564858 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565028 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565030 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.565106 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565040 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b2a4773-14f8-467d-a46f-a05b38375900-iptables-alerter-script\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565277 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565316 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565654 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw9hj\" (UniqueName: \"kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565719 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565803 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/679a9341-9428-4ae3-a14c-e72618082e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566124 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.565887 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566258 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5a7f4421-c460-4690-b755-a6fef7f2e487-metrics-server-audit-profiles\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566268 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566302 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566327 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566351 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566566 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/99c07bc5-c5be-4d09-9f96-1e50f37423be-metrics-client-ca\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566585 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566612 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566675 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566853 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5a7f4421-c460-4690-b755-a6fef7f2e487-client-ca-bundle\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.566947 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5a7f4421-c460-4690-b755-a6fef7f2e487-audit-log\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.567016 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/99c07bc5-c5be-4d09-9f96-1e50f37423be-secret-thanos-querier-tls\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.568183 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.567140 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/799484a3-d828-4795-b681-710baad6a32c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.583733 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.583702 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mwv7\" (UniqueName: \"kubernetes.io/projected/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-kube-api-access-8mwv7\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.583733 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.583757 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9qtb\" (UniqueName: \"kubernetes.io/projected/679a9341-9428-4ae3-a14c-e72618082e3b-kube-api-access-t9qtb\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.583733 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.583816 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wv2r\" (UniqueName: \"kubernetes.io/projected/dbe05ba0-4fc7-4df3-bf3b-af4feadb519d-kube-api-access-4wv2r\") pod \"openshift-state-metrics-7bd7489957-lvcvs\" (UID: \"dbe05ba0-4fc7-4df3-bf3b-af4feadb519d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-lvcvs" Oct 30 06:57:17.584048 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.583872 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd6nw\" (UniqueName: \"kubernetes.io/projected/799484a3-d828-4795-b681-710baad6a32c-kube-api-access-wd6nw\") pod \"prometheus-operator-9cd89b74c-qd27n\" (UID: \"799484a3-d828-4795-b681-710baad6a32c\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-qd27n" Oct 30 06:57:17.584048 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.583979 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"alertmanager-main-0\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:17.584168 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.584151 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bbc\" (UniqueName: \"kubernetes.io/projected/8b2a4773-14f8-467d-a46f-a05b38375900-kube-api-access-72bbc\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.584412 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.584385 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w2qb\" (UniqueName: \"kubernetes.io/projected/869e41d6-ed18-44a7-96ec-74efb259f79f-kube-api-access-6w2qb\") pod \"network-check-target-7lhl4\" (UID: \"869e41d6-ed18-44a7-96ec-74efb259f79f\") " pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:17.584517 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.584444 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj55v\" (UniqueName: \"kubernetes.io/projected/3122922f-c624-4694-9e9d-17893f6d9af8-kube-api-access-fj55v\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.584920 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.584904 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn9sn\" (UniqueName: \"kubernetes.io/projected/99c07bc5-c5be-4d09-9f96-1e50f37423be-kube-api-access-dn9sn\") pod \"thanos-querier-58f5c47444-cfzwr\" (UID: \"99c07bc5-c5be-4d09-9f96-1e50f37423be\") " pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:17.585770 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.585751 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbvw\" (UniqueName: \"kubernetes.io/projected/01a74e65-a10a-423a-b889-8105cdb97b1b-kube-api-access-brbvw\") pod \"ingress-canary-sm2hk\" (UID: \"01a74e65-a10a-423a-b889-8105cdb97b1b\") " pod="openshift-ingress-canary/ingress-canary-sm2hk" Oct 30 06:57:17.587153 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.587135 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwv82\" (UniqueName: \"kubernetes.io/projected/b1429333-ac51-49ee-8173-806b8ae7aeaf-kube-api-access-fwv82\") pod \"monitoring-plugin-77f4f76594-65fv7\" (UID: \"b1429333-ac51-49ee-8173-806b8ae7aeaf\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:17.587877 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.587859 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbkd2\" (UniqueName: \"kubernetes.io/projected/6215b7c5-298f-481a-b3ce-884ad7382b26-kube-api-access-zbkd2\") pod \"dns-default-55nz9\" (UID: \"6215b7c5-298f-481a-b3ce-884ad7382b26\") " pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:17.588200 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.588183 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzsb\" (UniqueName: \"kubernetes.io/projected/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kube-api-access-5jzsb\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.588333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.588317 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2vt\" (UniqueName: \"kubernetes.io/projected/20e60768-4e85-440c-8ddf-cb8f91fdd289-kube-api-access-fr2vt\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.588444 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.588424 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69kj\" (UniqueName: \"kubernetes.io/projected/77bfd548-5933-4191-a2b2-a9ec19692a83-kube-api-access-n69kj\") pod \"network-metrics-daemon-xgzmb\" (UID: \"77bfd548-5933-4191-a2b2-a9ec19692a83\") " pod="openshift-multus/network-metrics-daemon-xgzmb" Oct 30 06:57:17.588644 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.588615 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t8lp\" (UniqueName: \"kubernetes.io/projected/3fc44517-a592-4c5e-9a42-96047cabe380-kube-api-access-4t8lp\") pod \"kube-state-metrics-5dc8d7d89b-988bg\" (UID: \"3fc44517-a592-4c5e-9a42-96047cabe380\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-988bg" Oct 30 06:57:17.588971 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.588957 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx82n\" (UniqueName: \"kubernetes.io/projected/5a7f4421-c460-4690-b755-a6fef7f2e487-kube-api-access-rx82n\") pod \"metrics-server-65bd9d49b8-7bvx4\" (UID: \"5a7f4421-c460-4690-b755-a6fef7f2e487\") " pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:17.589277 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.589257 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"prometheus-k8s-0\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:17.589621 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.589606 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gvn9\" (UniqueName: \"kubernetes.io/projected/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-kube-api-access-8gvn9\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.590812 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.590794 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtb4k\" (UniqueName: \"kubernetes.io/projected/a87bd4fc-9395-46bf-87eb-13a159db993a-kube-api-access-wtb4k\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.592040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.592024 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw9hj\" (UniqueName: \"kubernetes.io/projected/9e4f6bc7-4192-4165-b16d-df0db49aedd9-kube-api-access-zw9hj\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.592468 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.592454 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmk2\" (UniqueName: \"kubernetes.io/projected/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-kube-api-access-wcmk2\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.667613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667510 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.667613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667555 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.667613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667572 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.667613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667590 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.667613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667608 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667625 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667664 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-netns\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667679 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b2a4773-14f8-467d-a46f-a05b38375900-host-slash\") pod \"iptables-alerter-94b4f\" (UID: \"8b2a4773-14f8-467d-a46f-a05b38375900\") " pod="openshift-network-operator/iptables-alerter-94b4f" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667670 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667710 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-kubernetes\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667719 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-socket-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667733 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667695 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-bin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667744 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-sys\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667752 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-sys-fs\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667762 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667796 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-device-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667800 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667832 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-conf-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667806 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-kubelet-config\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667844 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667880 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667862 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-kubelet\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667897 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667929 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667955 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667960 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f080f672-8cc7-4b00-a975-14ea5c5b1ae8-crio-socket\") pod \"insights-runtime-extractor-mwst9\" (UID: \"f080f672-8cc7-4b00-a975-14ea5c5b1ae8\") " pod="openshift-insights/insights-runtime-extractor-mwst9" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.667984 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-conf\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668006 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668011 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668006 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-run\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668001 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668026 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668052 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668057 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-cnibin\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668054 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ca9fe1e-35ee-469e-b4af-7c13480101f7-dbus\") pod \"global-pull-secret-syncer-5n2mb\" (UID: \"6ca9fe1e-35ee-469e-b4af-7c13480101f7\") " pod="kube-system/global-pull-secret-syncer-5n2mb" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668084 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668108 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668113 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-cnibin\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668085 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-bin\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668132 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-host\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668158 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668187 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-registration-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668187 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668207 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668211 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-k8s-cni-cncf-io\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668219 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-hostroot\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668228 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668245 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668277 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668285 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-modprobe-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668293 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668306 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-var-lib-kubelet\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668312 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668332 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668341 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-root\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668347 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-run-multus-certs\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668354 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/20e60768-4e85-440c-8ddf-cb8f91fdd289-host\") pod \"node-ca-rhhvm\" (UID: \"20e60768-4e85-440c-8ddf-cb8f91fdd289\") " pod="openshift-image-registry/node-ca-rhhvm" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668355 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668393 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-cni-netd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668310 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-etc-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668421 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668442 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysctl-d\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668449 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668449 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668479 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668479 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-sysconfig\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668482 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/679a9341-9428-4ae3-a14c-e72618082e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-kwcnj\" (UID: \"679a9341-9428-4ae3-a14c-e72618082e3b\") " pod="openshift-multus/multus-additional-cni-plugins-kwcnj" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668499 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668516 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668530 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668569 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668579 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-slash\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668581 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-node-exporter-wtmp\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668533 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-socket-dir-parent\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668597 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-etc-systemd\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668595 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668606 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-cni-multus\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668623 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-log-socket\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668660 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668669 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f0b0503e-7393-44db-8f5c-1aaaf2043b07-etc-selinux\") pod \"aws-ebs-csi-driver-node-lss5k\" (UID: \"f0b0503e-7393-44db-8f5c-1aaaf2043b07\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lss5k" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668685 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668705 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668729 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668740 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-host-var-lib-kubelet\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668729 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-systemd\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668757 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668762 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-systemd-units\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668776 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-netns\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668799 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668827 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668839 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-node-log\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668868 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec28222-b8cd-4a03-8a22-3a0d81e5b50f-sys\") pod \"node-exporter-82xqd\" (UID: \"3ec28222-b8cd-4a03-8a22-3a0d81e5b50f\") " pod="openshift-monitoring/node-exporter-82xqd" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668894 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668900 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-ovn\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668921 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668937 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668951 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668965 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-run-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.668972 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669011 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9e4f6bc7-4192-4165-b16d-df0db49aedd9-hosts-file\") pod \"node-resolver-wdkkw\" (UID: \"9e4f6bc7-4192-4165-b16d-df0db49aedd9\") " pod="openshift-dns/node-resolver-wdkkw" Oct 30 06:57:17.668973 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669013 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669040 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669061 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3122922f-c624-4694-9e9d-17893f6d9af8-lib-modules\") pod \"tuned-wzqxw\" (UID: \"3122922f-c624-4694-9e9d-17893f6d9af8\") " pod="openshift-cluster-node-tuning-operator/tuned-wzqxw" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669069 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669092 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669116 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-etc-kubernetes\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669130 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-var-lib-openvswitch\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669145 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a87bd4fc-9395-46bf-87eb-13a159db993a-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrvj8\" (UID: \"a87bd4fc-9395-46bf-87eb-13a159db993a\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669151 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-multus-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:17.671009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:17.669172 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8d4538a8-12ff-45c9-ae2c-fcc65413dae1-system-cni-dir\") pod \"multus-ff84l\" (UID: \"8d4538a8-12ff-45c9-ae2c-fcc65413dae1\") " pod="openshift-multus/multus-ff84l" Oct 30 06:57:18.221220 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:18.221187 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:18.285123 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:18.285098 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:18.348816 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:18.348774 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:18.546574 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:18.546548 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:18.565227 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:18.565201 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:19.377709 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:19.377680 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:19.504423 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:19.504396 5894 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 06:57:20.406992 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:20.406948 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:20.518919 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:20.518881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:20.548680 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:20.548650 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-55nz9" Oct 30 06:57:20.867436 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:20.867404 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-58f5c47444-cfzwr" Oct 30 06:57:21.199501 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:21.199425 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:21.308939 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:21.308905 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:22.267648 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:22.267594 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4js78" Oct 30 06:57:23.254280 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:23.254244 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.370552 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:23.370521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.946690 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:23.946662 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:24.086592 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:24.086556 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:25.921446 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:25.921418 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-65fv7" Oct 30 06:57:26.164074 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:26.164039 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:28.209719 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:28.209616 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:28.329120 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:28.329087 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:32.050117 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:32.050086 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrvj8" Oct 30 06:57:33.409130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:33.409072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:33.518571 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:33.518540 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:34.825496 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:34.825464 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 06:57:35.408151 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:35.408110 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:35.538662 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:35.538616 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:40.416145 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:40.416103 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:40.534259 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:40.534223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:42.927411 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:42.927384 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-65bd9d49b8-7bvx4" Oct 30 06:57:45.615522 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:45.615487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:45.738924 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:45.738893 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:46.166976 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:46.166946 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7lhl4" Oct 30 06:57:46.941535 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:46.941497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:47.061697 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:47.061655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:48.308663 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.308619 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:48.309076 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.308996 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="alertmanager" containerID="cri-o://235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c" gracePeriod=120 Oct 30 06:57:48.309076 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.309033 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-metric" containerID="cri-o://71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06" gracePeriod=120 Oct 30 06:57:48.309199 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.309096 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy" containerID="cri-o://0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67" gracePeriod=120 Oct 30 06:57:48.309199 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.309111 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="prom-label-proxy" containerID="cri-o://4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b" gracePeriod=120 Oct 30 06:57:48.309199 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.309050 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-web" containerID="cri-o://4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231" gracePeriod=120 Oct 30 06:57:48.309318 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.309068 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="config-reloader" containerID="cri-o://427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307" gracePeriod=120 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.619990 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b" exitCode=0 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620016 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06" exitCode=0 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620025 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67" exitCode=0 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620032 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307" exitCode=0 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620039 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c" exitCode=0 Oct 30 06:57:48.620075 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620057 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b"} Oct 30 06:57:48.620360 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620120 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06"} Oct 30 06:57:48.620360 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620146 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67"} Oct 30 06:57:48.620360 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620170 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307"} Oct 30 06:57:48.620360 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:48.620192 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c"} Oct 30 06:57:49.584854 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.584824 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:49.628119 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.628035 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df7d331-9233-4871-9d2f-6961c18ca460" containerID="4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231" exitCode=0 Oct 30 06:57:49.628273 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.628123 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:49.628273 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.628114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231"} Oct 30 06:57:49.628381 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.628309 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9df7d331-9233-4871-9d2f-6961c18ca460","Type":"ContainerDied","Data":"bf4628c04defbdc55086d14403ad459939fcce07cad76ac2b6ebb153ebf14405"} Oct 30 06:57:49.628381 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.628367 5894 scope.go:122] "RemoveContainer" containerID="4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b" Oct 30 06:57:49.635452 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635429 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635570 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635462 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635570 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635496 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635570 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635520 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635570 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635815 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635588 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635815 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635791 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635929 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635875 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.635929 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635906 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.636040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635938 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.636040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.635976 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.636040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.636005 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.636040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.636030 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:49.636040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.636036 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") pod \"9df7d331-9233-4871-9d2f-6961c18ca460\" (UID: \"9df7d331-9233-4871-9d2f-6961c18ca460\") " Oct 30 06:57:49.636353 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.636329 5894 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.637117 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.637086 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:49.637557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.637513 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:49.639559 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639514 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.639715 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639629 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "kube-api-access-vbn5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:49.639778 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639754 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.639861 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639838 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:49.639920 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639881 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.639977 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.639942 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.640187 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.640164 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:49.640255 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.640172 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.644038 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.643624 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.647262 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.647240 5894 scope.go:122] "RemoveContainer" containerID="71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06" Oct 30 06:57:49.651295 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.651272 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config" pod "9df7d331-9233-4871-9d2f-6961c18ca460" (UID: "9df7d331-9233-4871-9d2f-6961c18ca460"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:49.658962 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.658936 5894 scope.go:122] "RemoveContainer" containerID="0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67" Oct 30 06:57:49.665364 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.665344 5894 scope.go:122] "RemoveContainer" containerID="4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231" Oct 30 06:57:49.672125 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.672104 5894 scope.go:122] "RemoveContainer" containerID="427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307" Oct 30 06:57:49.678868 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.678849 5894 scope.go:122] "RemoveContainer" containerID="235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c" Oct 30 06:57:49.687861 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.687842 5894 scope.go:122] "RemoveContainer" containerID="08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e" Oct 30 06:57:49.696452 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.696425 5894 scope.go:122] "RemoveContainer" containerID="4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b" Oct 30 06:57:49.696976 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.696950 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b\": container with ID starting with 4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b not found: ID does not exist" containerID="4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b" Oct 30 06:57:49.697073 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.696989 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b"} err="failed to get container status \"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b\": rpc error: code = NotFound desc = could not find container \"4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b\": container with ID starting with 4cd8412ebb7fa3b8c4d45a6be2fb75a7fed59c627196ee224c3686e63e7fc11b not found: ID does not exist" Oct 30 06:57:49.697073 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697033 5894 scope.go:122] "RemoveContainer" containerID="71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06" Oct 30 06:57:49.697341 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.697318 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06\": container with ID starting with 71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06 not found: ID does not exist" containerID="71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06" Oct 30 06:57:49.697400 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697348 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06"} err="failed to get container status \"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06\": rpc error: code = NotFound desc = could not find container \"71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06\": container with ID starting with 71496f3e157cd7f43f6d2b9f0bab5328768be9df46cab0603982c62a94eccd06 not found: ID does not exist" Oct 30 06:57:49.697400 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697369 5894 scope.go:122] "RemoveContainer" containerID="0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67" Oct 30 06:57:49.697603 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.697584 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67\": container with ID starting with 0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67 not found: ID does not exist" containerID="0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67" Oct 30 06:57:49.697668 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697609 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67"} err="failed to get container status \"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67\": rpc error: code = NotFound desc = could not find container \"0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67\": container with ID starting with 0bbbce6230a45c624d50b376217b7f48342b9aee2174b8e8cfe77a29478e6a67 not found: ID does not exist" Oct 30 06:57:49.697736 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697627 5894 scope.go:122] "RemoveContainer" containerID="4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231" Oct 30 06:57:49.697995 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.697976 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231\": container with ID starting with 4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231 not found: ID does not exist" containerID="4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231" Oct 30 06:57:49.698091 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.697997 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231"} err="failed to get container status \"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231\": rpc error: code = NotFound desc = could not find container \"4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231\": container with ID starting with 4819783034f1319447ad294fbb85d119d85b77ae3dc66ef1906389427cc96231 not found: ID does not exist" Oct 30 06:57:49.698091 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698010 5894 scope.go:122] "RemoveContainer" containerID="427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307" Oct 30 06:57:49.698259 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.698237 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307\": container with ID starting with 427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307 not found: ID does not exist" containerID="427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307" Oct 30 06:57:49.698324 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698264 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307"} err="failed to get container status \"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307\": rpc error: code = NotFound desc = could not find container \"427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307\": container with ID starting with 427c305a3d187c2fb24a41bd74d1bf8397f81f255b261cf821f4630eccb18307 not found: ID does not exist" Oct 30 06:57:49.698324 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698279 5894 scope.go:122] "RemoveContainer" containerID="235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c" Oct 30 06:57:49.698526 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.698507 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c\": container with ID starting with 235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c not found: ID does not exist" containerID="235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c" Oct 30 06:57:49.698594 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698529 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c"} err="failed to get container status \"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c\": rpc error: code = NotFound desc = could not find container \"235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c\": container with ID starting with 235e838b27c81e8d7acabfa1d49e054fc9728467c67ff1f951a1039e78d1642c not found: ID does not exist" Oct 30 06:57:49.698594 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698547 5894 scope.go:122] "RemoveContainer" containerID="08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e" Oct 30 06:57:49.698851 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:49.698828 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e\": container with ID starting with 08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e not found: ID does not exist" containerID="08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e" Oct 30 06:57:49.698907 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.698857 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e"} err="failed to get container status \"08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e\": rpc error: code = NotFound desc = could not find container \"08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e\": container with ID starting with 08cb07011d1002e04136de7b680da25c624ad4fb96a78728ea619b98ef46b58e not found: ID does not exist" Oct 30 06:57:49.737586 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737558 5894 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-main-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737586 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737586 5894 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-config-out\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737598 5894 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-tls-assets\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737608 5894 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9df7d331-9233-4871-9d2f-6961c18ca460-alertmanager-main-db\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737616 5894 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-cluster-tls-config\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737624 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9df7d331-9233-4871-9d2f-6961c18ca460-metrics-client-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737656 5894 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-web-config\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737670 5894 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737682 5894 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737691 5894 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737699 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbn5z\" (UniqueName: \"kubernetes.io/projected/9df7d331-9233-4871-9d2f-6961c18ca460-kube-api-access-vbn5z\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.737761 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.737708 5894 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9df7d331-9233-4871-9d2f-6961c18ca460-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:49.978403 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.978367 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:49.992542 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:49.992512 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:50.014081 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014029 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:50.014353 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014339 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-metric" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014353 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-metric" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014363 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014368 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014381 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-web" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014386 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-web" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014398 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="config-reloader" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014403 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="config-reloader" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014412 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="alertmanager" Oct 30 06:57:50.014413 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014417 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="alertmanager" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014424 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="prom-label-proxy" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014429 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="prom-label-proxy" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014437 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="init-config-reloader" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014441 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="init-config-reloader" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014507 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="init-config-reloader" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014519 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-metric" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014525 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014531 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="kube-rbac-proxy-web" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014537 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="config-reloader" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014542 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="alertmanager" Oct 30 06:57:50.014858 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.014548 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" containerName="prom-label-proxy" Oct 30 06:57:50.051846 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.051810 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:50.052007 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.051941 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.069612 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069574 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 06:57:50.069813 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069666 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 06:57:50.069813 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069758 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 06:57:50.069813 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069808 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 06:57:50.069998 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069824 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 06:57:50.069998 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069846 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 06:57:50.069998 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069852 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 06:57:50.069998 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.069852 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cnr7r\"" Oct 30 06:57:50.070394 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.070380 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 06:57:50.084550 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.084531 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 06:57:50.141279 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141245 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141435 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141291 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141435 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141320 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8drj\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-kube-api-access-p8drj\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141435 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141390 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-out\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141549 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141490 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141549 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141525 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141619 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141588 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141619 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141610 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-web-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141716 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141630 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141764 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141741 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141764 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141756 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141838 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141772 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.141838 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.141790 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242555 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242599 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242653 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242617 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8drj\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-kube-api-access-p8drj\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242937 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242767 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-out\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242937 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242830 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242937 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242860 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.242937 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242908 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242937 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-web-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242965 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.242973 5894 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.243014 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.243038 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.243066 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243130 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.243094 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.243545 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.243519 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.244417 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.244151 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.245500 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.245470 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2b4020-74f0-41d7-b93f-61f90f6296f1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.246059 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.246041 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.246214 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.246191 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.246544 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.246521 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-web-config\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.246679 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.246661 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.247081 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.247047 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.247171 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.247063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.247232 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.247211 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.247484 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.247467 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-out\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.247801 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.247784 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6c2b4020-74f0-41d7-b93f-61f90f6296f1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.268374 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.268352 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8drj\" (UniqueName: \"kubernetes.io/projected/6c2b4020-74f0-41d7-b93f-61f90f6296f1-kube-api-access-p8drj\") pod \"alertmanager-main-0\" (UID: \"6c2b4020-74f0-41d7-b93f-61f90f6296f1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.406944 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.406909 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:50.428492 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.428465 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df7d331-9233-4871-9d2f-6961c18ca460" path="/var/lib/kubelet/pods/9df7d331-9233-4871-9d2f-6961c18ca460/volumes" Oct 30 06:57:50.457709 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.457677 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 06:57:50.517480 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.517444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:50.820773 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:50.820748 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 06:57:50.824932 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:50.824898 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2b4020_74f0_41d7_b93f_61f90f6296f1.slice/crio-1b68cb79128f6fe6c1213d080a805d7082ad23f4ff396b60a10893264568eb70 WatchSource:0}: Error finding container 1b68cb79128f6fe6c1213d080a805d7082ad23f4ff396b60a10893264568eb70: Status 404 returned error can't find the container with id 1b68cb79128f6fe6c1213d080a805d7082ad23f4ff396b60a10893264568eb70 Oct 30 06:57:51.650667 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:51.650600 5894 generic.go:356] "Generic (PLEG): container finished" podID="6c2b4020-74f0-41d7-b93f-61f90f6296f1" containerID="aa2a6c07ddf185c86684ab1e2cf15ddf2c75f0cf1b2eec89c9b59ff0b8fc9a9d" exitCode=0 Oct 30 06:57:51.650872 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:51.650697 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerDied","Data":"aa2a6c07ddf185c86684ab1e2cf15ddf2c75f0cf1b2eec89c9b59ff0b8fc9a9d"} Oct 30 06:57:51.650872 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:51.650858 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"1b68cb79128f6fe6c1213d080a805d7082ad23f4ff396b60a10893264568eb70"} Oct 30 06:57:52.230235 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.230156 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx"] Oct 30 06:57:52.232939 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.232901 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.250386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.250352 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 06:57:52.250386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.250352 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 06:57:52.250550 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.250398 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 06:57:52.250550 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.250502 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-nrldg\"" Oct 30 06:57:52.250550 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.250525 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 06:57:52.265613 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.265584 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx"] Oct 30 06:57:52.340503 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.340478 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 06:57:52.356453 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.356424 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 06:57:52.360321 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360298 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360408 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360340 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360476 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360450 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360510 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360486 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360547 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360510 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-metrics-client-ca\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360577 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360560 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxqqv\" (UniqueName: \"kubernetes.io/projected/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-kube-api-access-bxqqv\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360700 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360684 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-serving-certs-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.360740 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.360712 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-federate-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461173 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461138 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461173 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461175 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461425 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461196 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-metrics-client-ca\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461425 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461310 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxqqv\" (UniqueName: \"kubernetes.io/projected/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-kube-api-access-bxqqv\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461425 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461376 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-serving-certs-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461425 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461407 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-federate-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461610 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461456 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.461610 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.461500 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.462122 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.462083 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.462262 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.462090 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-metrics-client-ca\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.462336 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.462308 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-serving-certs-ca-bundle\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.463930 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.463907 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.464058 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.464037 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-secret-telemeter-client\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.464257 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.464240 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-federate-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.464702 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.464681 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-telemeter-client-tls\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.486872 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.486812 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxqqv\" (UniqueName: \"kubernetes.io/projected/d076bc5b-c1f1-499c-8f92-9bb0b3e922b5-kube-api-access-bxqqv\") pod \"telemeter-client-8d7c96fcc-ltvtx\" (UID: \"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5\") " pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.546549 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.546515 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" Oct 30 06:57:52.658404 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658374 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"9bebe666c0d32f3349477113edfca1ea6fd127f3c0891513964007ae4de074db"} Oct 30 06:57:52.658565 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658440 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"4f32a288d6f419332edb591c70cdc53bca5860aae6c53ed19d14bb2d5ca3814a"} Oct 30 06:57:52.658565 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658474 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"21fddfe1416e07d4ad5fffd64fec8fbacfb3519a46adcc51bb3b67e35d85fc76"} Oct 30 06:57:52.658565 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658507 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"4b2ea4a93ca0a436796133709926a29becf7d2bd59e2ebae7ae146305b9cf799"} Oct 30 06:57:52.658565 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658533 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"250111265aee1b8f7900e93252e6654e2deee1eef87cd634b7183d711443053d"} Oct 30 06:57:52.658565 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.658559 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6c2b4020-74f0-41d7-b93f-61f90f6296f1","Type":"ContainerStarted","Data":"b017c522e44224054d700733ceecdc241588549e02485bde80deec07e655d233"} Oct 30 06:57:52.720744 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.720692 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.709428365 podStartE2EDuration="3.720676346s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.247981ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:52.719441372 +0000 UTC m=+36.410358848" watchObservedRunningTime="2026-10-30 06:57:52.720676346 +0000 UTC m=+36.411593780" Oct 30 06:57:52.865983 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.865955 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx"] Oct 30 06:57:52.867381 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:52.867356 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd076bc5b_c1f1_499c_8f92_9bb0b3e922b5.slice/crio-6759d3fdc943acc8173790f53658013c746195f4fab099c7bbd0748ed4586d8e WatchSource:0}: Error finding container 6759d3fdc943acc8173790f53658013c746195f4fab099c7bbd0748ed4586d8e: Status 404 returned error can't find the container with id 6759d3fdc943acc8173790f53658013c746195f4fab099c7bbd0748ed4586d8e Oct 30 06:57:52.869214 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.869198 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:57:52.948231 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948191 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:52.948557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948487 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="prometheus" containerID="cri-o://ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" gracePeriod=600 Oct 30 06:57:52.948557 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948526 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="thanos-sidecar" containerID="cri-o://67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352" gracePeriod=600 Oct 30 06:57:52.948737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948536 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="config-reloader" containerID="cri-o://11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b" gracePeriod=600 Oct 30 06:57:52.948737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948543 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-web" containerID="cri-o://dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de" gracePeriod=600 Oct 30 06:57:52.948737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948513 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy" containerID="cri-o://d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6" gracePeriod=600 Oct 30 06:57:52.948737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:52.948536 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e" gracePeriod=600 Oct 30 06:57:53.565332 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:53.565285 5894 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 06:57:53.565829 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:53.565355 5894 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 06:57:53.565829 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:53.565375 5894 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 30 06:57:53.565829 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.565390 5894 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 30 06:57:53.667554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667466 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e" exitCode=0 Oct 30 06:57:53.667554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667499 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6" exitCode=0 Oct 30 06:57:53.667554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667509 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352" exitCode=0 Oct 30 06:57:53.667554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667522 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b" exitCode=0 Oct 30 06:57:53.667554 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667537 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" exitCode=0 Oct 30 06:57:53.667910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667548 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e"} Oct 30 06:57:53.667910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667621 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6"} Oct 30 06:57:53.667910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667670 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352"} Oct 30 06:57:53.667910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667712 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b"} Oct 30 06:57:53.667910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.667754 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888"} Oct 30 06:57:53.668976 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:53.668950 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" event={"ID":"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5","Type":"ContainerStarted","Data":"6759d3fdc943acc8173790f53658013c746195f4fab099c7bbd0748ed4586d8e"} Oct 30 06:57:54.203296 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.203267 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.279782 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.279750 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.279968 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.279793 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.279968 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.279829 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.279968 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.279878 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280112 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280082 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.280576 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280226 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280576 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280274 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280576 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280301 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280576 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280501 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280690 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280747 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280768 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280795 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280820 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280855 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280879 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280884 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.280957 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280948 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.280990 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281073 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281125 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281177 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281223 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") pod \"6daadbca-3412-44d6-85ab-94e80b0ca09a\" (UID: \"6daadbca-3412-44d6-85ab-94e80b0ca09a\") " Oct 30 06:57:54.281458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281357 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.281812 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281529 5894 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.281812 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281549 5894 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-metrics-client-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.281812 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281565 5894 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.281812 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.281580 5894 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.282445 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.282500 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.283117 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.283548 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.283722 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:54.284094 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.283722 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:54.284553 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.284526 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.285802 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.285761 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.286558 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286498 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "kube-api-access-fz4fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:54.286558 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286517 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.286558 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286532 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:54.286883 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286846 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.286953 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286901 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.287019 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.286995 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.287820 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.287784 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.300910 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.300876 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config" pod "6daadbca-3412-44d6-85ab-94e80b0ca09a" (UID: "6daadbca-3412-44d6-85ab-94e80b0ca09a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:54.382152 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382120 5894 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-web-config\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382152 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382156 5894 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382170 5894 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-grpc-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382183 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz4fh\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-kube-api-access-fz4fh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382198 5894 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382210 5894 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-metrics-client-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382225 5894 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382249 5894 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-kube-rbac-proxy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382263 5894 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-db\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382276 5894 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382291 5894 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382303 5894 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6daadbca-3412-44d6-85ab-94e80b0ca09a-config-out\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.382333 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382317 5894 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.383168 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382490 5894 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6daadbca-3412-44d6-85ab-94e80b0ca09a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.383168 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382518 5894 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6daadbca-3412-44d6-85ab-94e80b0ca09a-tls-assets\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.383168 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.382535 5894 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6daadbca-3412-44d6-85ab-94e80b0ca09a-config\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 06:57:54.677228 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.677197 5894 generic.go:356] "Generic (PLEG): container finished" podID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerID="dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de" exitCode=0 Oct 30 06:57:54.677556 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.677279 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de"} Oct 30 06:57:54.677556 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.677298 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.677556 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.677424 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6daadbca-3412-44d6-85ab-94e80b0ca09a","Type":"ContainerDied","Data":"2d00763a83420b6c9f01b3b5003b57bb77edfb5c350dabfb7477811de55024b9"} Oct 30 06:57:54.677556 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.677475 5894 scope.go:122] "RemoveContainer" containerID="6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e" Oct 30 06:57:54.679532 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.679498 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" event={"ID":"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5","Type":"ContainerStarted","Data":"83657c4acfdde04fddeca407816af252a017b7cc2b23fadaf3db28f358860000"} Oct 30 06:57:54.679690 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.679560 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" event={"ID":"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5","Type":"ContainerStarted","Data":"61efd33b27196c0f82ed390f8139d071ca86d21123eb1be1368ff9b641ed9216"} Oct 30 06:57:54.687696 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.686502 5894 scope.go:122] "RemoveContainer" containerID="d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6" Oct 30 06:57:54.696016 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.695994 5894 scope.go:122] "RemoveContainer" containerID="dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de" Oct 30 06:57:54.713365 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.713345 5894 scope.go:122] "RemoveContainer" containerID="67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352" Oct 30 06:57:54.726921 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.726885 5894 scope.go:122] "RemoveContainer" containerID="11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b" Oct 30 06:57:54.734364 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.734344 5894 scope.go:122] "RemoveContainer" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" Oct 30 06:57:54.742078 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.742056 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:54.743346 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.743329 5894 scope.go:122] "RemoveContainer" containerID="8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028" Oct 30 06:57:54.750549 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.750528 5894 scope.go:122] "RemoveContainer" containerID="6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e" Oct 30 06:57:54.750859 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.750839 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e\": container with ID starting with 6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e not found: ID does not exist" containerID="6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e" Oct 30 06:57:54.750931 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.750867 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e"} err="failed to get container status \"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e\": rpc error: code = NotFound desc = could not find container \"6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e\": container with ID starting with 6534d24ff032dea50acfb48a9e78eb779e319e1a5c0f46f0b98545fbcc2e2d0e not found: ID does not exist" Oct 30 06:57:54.750931 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.750886 5894 scope.go:122] "RemoveContainer" containerID="d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6" Oct 30 06:57:54.751162 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.751144 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6\": container with ID starting with d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6 not found: ID does not exist" containerID="d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6" Oct 30 06:57:54.751205 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.751169 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6"} err="failed to get container status \"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6\": rpc error: code = NotFound desc = could not find container \"d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6\": container with ID starting with d86ffc8e1b2161165cee556d969c5db8750534256a5c89de11bb6535e9680be6 not found: ID does not exist" Oct 30 06:57:54.751205 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.751191 5894 scope.go:122] "RemoveContainer" containerID="dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de" Oct 30 06:57:54.751521 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.751486 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de\": container with ID starting with dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de not found: ID does not exist" containerID="dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de" Oct 30 06:57:54.751587 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.751529 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de"} err="failed to get container status \"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de\": rpc error: code = NotFound desc = could not find container \"dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de\": container with ID starting with dcabbc706d3b0df7cc7a43a4d0a5fa72895755694d2b8980ef850ae6db8df4de not found: ID does not exist" Oct 30 06:57:54.751587 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.751551 5894 scope.go:122] "RemoveContainer" containerID="67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352" Oct 30 06:57:54.752099 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.752077 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352\": container with ID starting with 67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352 not found: ID does not exist" containerID="67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352" Oct 30 06:57:54.752201 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.752106 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352"} err="failed to get container status \"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352\": rpc error: code = NotFound desc = could not find container \"67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352\": container with ID starting with 67f81b67547a1a72c65a43d267e1c9b9b4f29856d3547c0d4abd761c2e77d352 not found: ID does not exist" Oct 30 06:57:54.752201 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.752126 5894 scope.go:122] "RemoveContainer" containerID="11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b" Oct 30 06:57:54.752471 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.752437 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b\": container with ID starting with 11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b not found: ID does not exist" containerID="11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b" Oct 30 06:57:54.752523 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.752480 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b"} err="failed to get container status \"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b\": rpc error: code = NotFound desc = could not find container \"11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b\": container with ID starting with 11514a2ba45af6008127f481a8f1ffa0d6267ed6d963b31a5c6d98766cd9734b not found: ID does not exist" Oct 30 06:57:54.752523 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.752499 5894 scope.go:122] "RemoveContainer" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" Oct 30 06:57:54.753005 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.752987 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888\": container with ID starting with ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888 not found: ID does not exist" containerID="ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888" Oct 30 06:57:54.753055 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.753009 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888"} err="failed to get container status \"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888\": rpc error: code = NotFound desc = could not find container \"ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888\": container with ID starting with ca49b5698756857cee0e761a61d8374d406fc7e9c4889ede29432859c3697888 not found: ID does not exist" Oct 30 06:57:54.753055 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.753021 5894 scope.go:122] "RemoveContainer" containerID="8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028" Oct 30 06:57:54.753261 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:57:54.753247 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028\": container with ID starting with 8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028 not found: ID does not exist" containerID="8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028" Oct 30 06:57:54.753301 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.753263 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028"} err="failed to get container status \"8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028\": rpc error: code = NotFound desc = could not find container \"8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028\": container with ID starting with 8c63cb3e3146cb10b1e8c949e31484838bc96504de15adb6632e2e1a22410028 not found: ID does not exist" Oct 30 06:57:54.756831 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.756811 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:54.776573 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776539 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:54.776932 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776915 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776935 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776954 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="init-config-reloader" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776962 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="init-config-reloader" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776981 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-thanos" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.776989 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-thanos" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777001 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="config-reloader" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777008 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="config-reloader" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777021 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-web" Oct 30 06:57:54.777033 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777030 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-web" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777050 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="prometheus" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777058 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="prometheus" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777071 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="thanos-sidecar" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777080 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="thanos-sidecar" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777177 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777190 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="init-config-reloader" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777203 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-thanos" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777211 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="config-reloader" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777219 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="kube-rbac-proxy-web" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777232 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="prometheus" Oct 30 06:57:54.777448 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.777241 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" containerName="thanos-sidecar" Oct 30 06:57:54.782891 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.782867 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.801777 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.801695 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 06:57:54.801777 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.801716 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 06:57:54.801777 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.801731 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 06:57:54.801777 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.801733 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 06:57:54.802126 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.801988 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 06:57:54.802126 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802022 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802235 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802265 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802272 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6pfvt\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802310 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802319 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-66t9c9j0f6gnq\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802299 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 06:57:54.802343 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802330 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 06:57:54.802611 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.802346 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 06:57:54.818145 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.818071 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 06:57:54.818145 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.818081 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 06:57:54.818924 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.818900 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:54.885725 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885691 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885725 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885723 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885969 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885744 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885969 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885817 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885969 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885870 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885969 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885899 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.885969 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885960 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.885984 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886014 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886059 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886084 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886179 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886161 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886182 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886201 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886214 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwmfh\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-kube-api-access-rwmfh\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886257 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886310 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886356 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886386 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886380 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.886669 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.886436 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987628 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987591 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987628 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987656 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987684 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987707 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwmfh\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-kube-api-access-rwmfh\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987734 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987772 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987797 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987830 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987857 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.987903 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987888 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987912 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987941 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.987975 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988019 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988046 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988079 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988103 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988133 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988170 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988283 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988195 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988805 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988729 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.988874 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.988801 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.990684 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.989406 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.990684 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.989772 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.990684 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.990093 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.990684 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.990606 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.990991 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.990950 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.991275 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.991249 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.991420 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.991365 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.991827 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.991781 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.992009 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.991981 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddf96c98-f625-41ad-a303-63b2c4d92e95-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993057 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993022 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993274 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993246 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-config\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993342 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993303 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993342 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993307 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993489 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993465 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddf96c98-f625-41ad-a303-63b2c4d92e95-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.993674 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.993651 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.994195 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.994164 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:54.994432 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:54.994417 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddf96c98-f625-41ad-a303-63b2c4d92e95-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:55.013364 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.013341 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwmfh\" (UniqueName: \"kubernetes.io/projected/ddf96c98-f625-41ad-a303-63b2c4d92e95-kube-api-access-rwmfh\") pod \"prometheus-k8s-0\" (UID: \"ddf96c98-f625-41ad-a303-63b2c4d92e95\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:55.105624 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.105539 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:57:55.441466 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.441439 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 06:57:55.442423 ip-10-0-143-90 kubenswrapper[5894]: W1030 06:57:55.442386 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf96c98_f625_41ad_a303_63b2c4d92e95.slice/crio-93b1584fde57b90e8391524ce1081527aca851d7d08ddd39e57f1aaab7f58520 WatchSource:0}: Error finding container 93b1584fde57b90e8391524ce1081527aca851d7d08ddd39e57f1aaab7f58520: Status 404 returned error can't find the container with id 93b1584fde57b90e8391524ce1081527aca851d7d08ddd39e57f1aaab7f58520 Oct 30 06:57:55.686108 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.686069 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" event={"ID":"d076bc5b-c1f1-499c-8f92-9bb0b3e922b5","Type":"ContainerStarted","Data":"71b6f047154c23ebb5bea0c25409003d03eba637cda381cbf8715ddc36522d09"} Oct 30 06:57:55.687737 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.687711 5894 generic.go:356] "Generic (PLEG): container finished" podID="ddf96c98-f625-41ad-a303-63b2c4d92e95" containerID="63ce6e6b41c4d845f2427d657df7c8d8666b7b77981002ee19a188095e17b292" exitCode=0 Oct 30 06:57:55.687882 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.687795 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerDied","Data":"63ce6e6b41c4d845f2427d657df7c8d8666b7b77981002ee19a188095e17b292"} Oct 30 06:57:55.687882 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.687857 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"93b1584fde57b90e8391524ce1081527aca851d7d08ddd39e57f1aaab7f58520"} Oct 30 06:57:55.737815 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:55.737767 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8d7c96fcc-ltvtx" podStartSLOduration=2.047460942 podStartE2EDuration="3.737754439s" totalImagesPullingTime="1.690293497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:55.737042701 +0000 UTC m=+39.427960138" watchObservedRunningTime="2026-10-30 06:57:55.737754439 +0000 UTC m=+39.428671875" Oct 30 06:57:56.052423 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.052389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.364097 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.364064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.428990 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.428952 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6daadbca-3412-44d6-85ab-94e80b0ca09a" path="/var/lib/kubelet/pods/6daadbca-3412-44d6-85ab-94e80b0ca09a/volumes" Oct 30 06:57:56.482027 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.481988 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.696048 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.695958 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"6398746348ea01ce6ee9c91c487ba53f294c4719759385990d18750d7ed8ee4e"} Oct 30 06:57:56.696048 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.696047 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"68db45bbc3be293ca03965b381d27d9c3aee48ae1f401c1d3907ccd0eac51ef0"} Oct 30 06:57:56.696524 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.696076 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"7f6e83d32a9ec299e431314de51eef2f9a2e3d981cf27032172849c47cedb0e0"} Oct 30 06:57:56.696524 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.696104 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"914f004e295a33228424ffe7eec694d844875a04ead33406cf3faf63be832dd0"} Oct 30 06:57:56.696524 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.696133 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"4d080ec374df274b8d94719fc324591c500f7d8c19c5d0fe3313011a04f544bd"} Oct 30 06:57:56.696524 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.696156 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddf96c98-f625-41ad-a303-63b2c4d92e95","Type":"ContainerStarted","Data":"e33f74248ae7b5c12ff78a8e14b6bf68e7efdbff84459778de2e7094153068e6"} Oct 30 06:57:56.771468 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:57:56.771406 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.753996494 podStartE2EDuration="2.77139377s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.397276ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:57:56.769721645 +0000 UTC m=+40.460639083" watchObservedRunningTime="2026-10-30 06:57:56.77139377 +0000 UTC m=+40.462311204" Oct 30 06:58:00.106738 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:00.106691 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:58:04.825673 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:58:04.825622 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 06:58:05.407837 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:05.407792 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:05.520698 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:05.520661 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:24.080178 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:24.080144 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:34.826070 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:58:34.826027 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 06:58:43.087791 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:43.087747 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:47.588970 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:47.588931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:54.453588 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:54.453548 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:58:54.469728 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:54.469703 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:58:54.928310 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:58:54.928281 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 06:59:02.845504 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:02.845411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:02.965705 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:02.965665 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:04.825378 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:59:04.825346 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 06:59:10.639681 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:10.639628 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:10.766040 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:10.766006 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.299250 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:12.299216 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.417230 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:12.417198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:15.521458 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:15.521420 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:15.633850 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:15.633817 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.413506 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:20.413471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.540568 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:20.540530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:22.603380 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:22.603337 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:22.716902 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:22.716867 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.191354 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:24.191314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.699941 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:24.699901 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.824013 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:24.823977 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:34.826771 ip-10-0-143-90 kubenswrapper[5894]: E1030 06:59:34.826733 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 06:59:40.603702 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:40.603664 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:50.918908 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:50.918862 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:51.038864 ip-10-0-143-90 kubenswrapper[5894]: I1030 06:59:51.038830 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:04.826185 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:00:04.826148 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:00:24.313492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:00:24.313448 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:34.825940 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:00:34.825855 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:01:04.828013 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:01:04.827942 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:01:18.573354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:18.573318 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:19.710025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:19.709984 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:24.455879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:24.455846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:33.024950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:33.024917 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:33.200370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:33.200335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:33.323664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:33.323604 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:34.609200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:34.609169 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:34.735499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:34.735467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:34.826689 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:01:34.826652 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:01:35.867054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:35.867008 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:38.467774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:38.467738 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:38.594183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:38.594129 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:39.847072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:39.847035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:39.974676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:39.974624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:40.089167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:40.089133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:43.050962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:01:43.050921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:04.826655 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:02:04.826557 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:02:19.951303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:19.951258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:20.228880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:20.228793 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:20.880741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:20.880702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:21.003323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.003289 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz"] Oct 30 07:02:21.006704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.006686 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.046179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.043080 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz"] Oct 30 07:02:21.099338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099306 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 07:02:21.099524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099383 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 07:02:21.099524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099446 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-62jmt\"" Oct 30 07:02:21.099524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099455 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 07:02:21.099524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099474 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 07:02:21.099524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.099504 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 07:02:21.139504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.139429 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:21.143423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143398 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.143534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143437 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.143534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143459 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.143613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143538 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.143668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143617 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75snw\" (UniqueName: \"kubernetes.io/projected/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-kube-api-access-75snw\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.143738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.143725 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244320 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244418 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244448 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75snw\" (UniqueName: \"kubernetes.io/projected/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-kube-api-access-75snw\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244528 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.244962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.244907 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-manifests\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.246989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.246957 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-tmp\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.247096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.247027 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-metrics-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.247181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.247164 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-webhook-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.247247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.247181 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.274174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.274139 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75snw\" (UniqueName: \"kubernetes.io/projected/818a14bd-247f-4a04-bb3e-19fa57d9e4a2-kube-api-access-75snw\") pod \"opendatahub-operator-controller-manager-77996896cc-hslfz\" (UID: \"818a14bd-247f-4a04-bb3e-19fa57d9e4a2\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.311410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.311379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:21.321788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.321764 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:21.449848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.449742 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:21.676157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:21.676132 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz"] Oct 30 07:02:21.678409 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:21.678384 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818a14bd_247f_4a04_bb3e_19fa57d9e4a2.slice/crio-0d56267f866903eb47e695ec5e336b77d957c7163ed17455f70d7a6fb8a5913c WatchSource:0}: Error finding container 0d56267f866903eb47e695ec5e336b77d957c7163ed17455f70d7a6fb8a5913c: Status 404 returned error can't find the container with id 0d56267f866903eb47e695ec5e336b77d957c7163ed17455f70d7a6fb8a5913c Oct 30 07:02:22.383435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:22.383383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:22.656469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:22.656319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:22.664831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:22.664792 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" event={"ID":"818a14bd-247f-4a04-bb3e-19fa57d9e4a2","Type":"ContainerStarted","Data":"0d56267f866903eb47e695ec5e336b77d957c7163ed17455f70d7a6fb8a5913c"} Oct 30 07:02:23.717532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:23.717495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:24.570157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:24.570120 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:24.674702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:24.674669 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" event={"ID":"818a14bd-247f-4a04-bb3e-19fa57d9e4a2","Type":"ContainerStarted","Data":"7c6b5cfc05d6d00bc03cc9f82e15d6fb9f1fedeceba9082c517185818ad9ffe1"} Oct 30 07:02:24.742336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:24.742252 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:24.889766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:24.889734 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:25.050095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:25.050061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:25.680067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:25.680038 5894 generic.go:356] "Generic (PLEG): container finished" podID="818a14bd-247f-4a04-bb3e-19fa57d9e4a2" containerID="7c6b5cfc05d6d00bc03cc9f82e15d6fb9f1fedeceba9082c517185818ad9ffe1" exitCode=0 Oct 30 07:02:25.680249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:25.680121 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" event={"ID":"818a14bd-247f-4a04-bb3e-19fa57d9e4a2","Type":"ContainerDied","Data":"7c6b5cfc05d6d00bc03cc9f82e15d6fb9f1fedeceba9082c517185818ad9ffe1"} Oct 30 07:02:26.685697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:26.685660 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" event={"ID":"818a14bd-247f-4a04-bb3e-19fa57d9e4a2","Type":"ContainerStarted","Data":"43d6bad10bac973550a142bbfb8da3db50ed53238cee02d8d49e9117b234138c"} Oct 30 07:02:26.686130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:26.685767 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:26.780229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:26.780167 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" podStartSLOduration=3.074942732 podStartE2EDuration="6.780150421s" totalImagesPullingTime="3.157855426s" totalInitContainerRuntime="547.352263ms" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:26.765112348 +0000 UTC m=+310.456029799" watchObservedRunningTime="2026-10-30 07:02:26.780150421 +0000 UTC m=+310.471067941" Oct 30 07:02:30.319154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:30.319118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:30.558247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:30.558203 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:30.681781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:30.681691 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:33.411721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:33.411685 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:33.526325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:33.526286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:33.931992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:33.931958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:34.737691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:34.737655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:34.826200 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:02:34.826170 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:02:35.532651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:35.532596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:36.285073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.285039 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn"] Oct 30 07:02:36.291976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.291948 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.307843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.307814 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn"] Oct 30 07:02:36.321386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.321359 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 07:02:36.321386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.321387 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 07:02:36.325411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.325386 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-qlwxr\"" Oct 30 07:02:36.380967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.380927 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-tmp\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.380967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.380973 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5752r\" (UniqueName: \"kubernetes.io/projected/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-kube-api-access-5752r\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.482117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.482078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-tmp\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.482117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.482116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5752r\" (UniqueName: \"kubernetes.io/projected/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-kube-api-access-5752r\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.482481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.482449 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-tmp\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.507064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.507029 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5752r\" (UniqueName: \"kubernetes.io/projected/a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79-kube-api-access-5752r\") pod \"jobset-operator-674cd59dc5-82rhn\" (UID: \"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.618342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.618258 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" Oct 30 07:02:36.922006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:36.921978 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn"] Oct 30 07:02:36.923671 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:36.923617 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2dd47ff_75d4_490d_b1a1_f4bb5b9bbf79.slice/crio-57ba1ff99132a238b6b86c6f64a517d76b6d627741c2e6aa5b4c72011e42e1b9 WatchSource:0}: Error finding container 57ba1ff99132a238b6b86c6f64a517d76b6d627741c2e6aa5b4c72011e42e1b9: Status 404 returned error can't find the container with id 57ba1ff99132a238b6b86c6f64a517d76b6d627741c2e6aa5b4c72011e42e1b9 Oct 30 07:02:37.692601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:37.692572 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-77996896cc-hslfz" Oct 30 07:02:37.729386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:37.729317 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" event={"ID":"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79","Type":"ContainerStarted","Data":"57ba1ff99132a238b6b86c6f64a517d76b6d627741c2e6aa5b4c72011e42e1b9"} Oct 30 07:02:38.103014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:38.102981 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:39.740178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:39.740143 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" event={"ID":"a2dd47ff-75d4-490d-b1a1-f4bb5b9bbf79","Type":"ContainerStarted","Data":"3f067462271fbf3c66c98f5137ead8fab4cca8195227984e9c6e0ab454baa9e2"} Oct 30 07:02:39.790090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:39.790029 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-82rhn" podStartSLOduration=1.106400047 podStartE2EDuration="3.790010355s" totalImagesPullingTime="2.683610308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:39.789630392 +0000 UTC m=+323.480547831" watchObservedRunningTime="2026-10-30 07:02:39.790010355 +0000 UTC m=+323.480927798" Oct 30 07:02:41.925043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:41.925003 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:42.050737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.050701 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj"] Oct 30 07:02:42.053094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.053077 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.064721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.064696 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj"] Oct 30 07:02:42.078307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.078278 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 07:02:42.078307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.078299 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k9xxd\"" Oct 30 07:02:42.078489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.078279 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 07:02:42.138803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.138763 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq86v\" (UniqueName: \"kubernetes.io/projected/9a70a327-53bd-4c9a-bc35-77edcf45e595-kube-api-access-tq86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.138803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.138808 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70a327-53bd-4c9a-bc35-77edcf45e595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.240096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.240011 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq86v\" (UniqueName: \"kubernetes.io/projected/9a70a327-53bd-4c9a-bc35-77edcf45e595-kube-api-access-tq86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.240096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.240065 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70a327-53bd-4c9a-bc35-77edcf45e595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.240490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.240471 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70a327-53bd-4c9a-bc35-77edcf45e595-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.265250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.265219 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq86v\" (UniqueName: \"kubernetes.io/projected/9a70a327-53bd-4c9a-bc35-77edcf45e595-kube-api-access-tq86v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-whjwj\" (UID: \"9a70a327-53bd-4c9a-bc35-77edcf45e595\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.363486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.363450 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" Oct 30 07:02:42.692044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.692021 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj"] Oct 30 07:02:42.694157 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:42.694128 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70a327_53bd_4c9a_bc35_77edcf45e595.slice/crio-fe6541264f3646ec90163180bd693771be209358e04dcaf199295dcd905d4e35 WatchSource:0}: Error finding container fe6541264f3646ec90163180bd693771be209358e04dcaf199295dcd905d4e35: Status 404 returned error can't find the container with id fe6541264f3646ec90163180bd693771be209358e04dcaf199295dcd905d4e35 Oct 30 07:02:42.753394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:42.753361 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" event={"ID":"9a70a327-53bd-4c9a-bc35-77edcf45e595","Type":"ContainerStarted","Data":"fe6541264f3646ec90163180bd693771be209358e04dcaf199295dcd905d4e35"} Oct 30 07:02:44.649689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.648066 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx"] Oct 30 07:02:44.657098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.657048 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:44.688621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.688557 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 07:02:44.688621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.688553 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g6rxj\"" Oct 30 07:02:44.688621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.688565 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 07:02:44.688621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.688612 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 07:02:44.689527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.689500 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx"] Oct 30 07:02:44.769264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.769231 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqhqd\" (UniqueName: \"kubernetes.io/projected/01b69050-e4b3-483e-8065-0ef33a849871-kube-api-access-rqhqd\") pod \"dns-operator-controller-manager-8557cbdb84-5c2cx\" (UID: \"01b69050-e4b3-483e-8065-0ef33a849871\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:44.870767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.870721 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqhqd\" (UniqueName: \"kubernetes.io/projected/01b69050-e4b3-483e-8065-0ef33a849871-kube-api-access-rqhqd\") pod \"dns-operator-controller-manager-8557cbdb84-5c2cx\" (UID: \"01b69050-e4b3-483e-8065-0ef33a849871\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:44.896693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.896623 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqhqd\" (UniqueName: \"kubernetes.io/projected/01b69050-e4b3-483e-8065-0ef33a849871-kube-api-access-rqhqd\") pod \"dns-operator-controller-manager-8557cbdb84-5c2cx\" (UID: \"01b69050-e4b3-483e-8065-0ef33a849871\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:44.973802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:44.973709 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:45.287063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:45.287042 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx"] Oct 30 07:02:45.288919 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:45.288890 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b69050_e4b3_483e_8065_0ef33a849871.slice/crio-5f1fcae6e6172851838ccd4d58241667dce56fa3027805eeabaeadfbae8610da WatchSource:0}: Error finding container 5f1fcae6e6172851838ccd4d58241667dce56fa3027805eeabaeadfbae8610da: Status 404 returned error can't find the container with id 5f1fcae6e6172851838ccd4d58241667dce56fa3027805eeabaeadfbae8610da Oct 30 07:02:45.769098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:45.769055 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" event={"ID":"9a70a327-53bd-4c9a-bc35-77edcf45e595","Type":"ContainerStarted","Data":"987ea1314254762fd70670687c3a28f170b8faead3313c0dd32b587667c38060"} Oct 30 07:02:45.770380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:45.770355 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" event={"ID":"01b69050-e4b3-483e-8065-0ef33a849871","Type":"ContainerStarted","Data":"5f1fcae6e6172851838ccd4d58241667dce56fa3027805eeabaeadfbae8610da"} Oct 30 07:02:45.849825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:45.849771 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-whjwj" podStartSLOduration=1.3261830030000001 podStartE2EDuration="3.849755307s" totalImagesPullingTime="2.523572304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:45.834949029 +0000 UTC m=+329.525866467" watchObservedRunningTime="2026-10-30 07:02:45.849755307 +0000 UTC m=+329.540672744" Oct 30 07:02:47.329133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.329090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:47.514338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.514298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:47.638014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.637944 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5"] Oct 30 07:02:47.648948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.648913 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.667195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.667158 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5"] Oct 30 07:02:47.680078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.680026 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 07:02:47.682739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.682713 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 07:02:47.682739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.682736 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mvbsd\"" Oct 30 07:02:47.700132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.699987 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.700132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.700031 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnwc\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-kube-api-access-gsnwc\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.801200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.801160 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.801388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.801207 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnwc\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-kube-api-access-gsnwc\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.826617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.826584 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.826808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.826598 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnwc\" (UniqueName: \"kubernetes.io/projected/7272d9f0-29bb-4254-8580-850653c638a0-kube-api-access-gsnwc\") pod \"cert-manager-webhook-5dbbbf75cc-b4mn5\" (UID: \"7272d9f0-29bb-4254-8580-850653c638a0\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:47.965945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:47.965858 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:48.155337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.155307 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb"] Oct 30 07:02:48.158593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.158574 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.180931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.180899 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb"] Oct 30 07:02:48.186996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.186968 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:48.192983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.192953 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 07:02:48.193083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.192986 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5hg8w\"" Oct 30 07:02:48.193083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.192986 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 07:02:48.270161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.270137 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5"] Oct 30 07:02:48.271451 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:48.271425 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7272d9f0_29bb_4254_8580_850653c638a0.slice/crio-1fce4483bf61e64169ed863a6cc836b9fa541fee1ac6fd72da860b3a5213fe3e WatchSource:0}: Error finding container 1fce4483bf61e64169ed863a6cc836b9fa541fee1ac6fd72da860b3a5213fe3e: Status 404 returned error can't find the container with id 1fce4483bf61e64169ed863a6cc836b9fa541fee1ac6fd72da860b3a5213fe3e Oct 30 07:02:48.307025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.306990 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4864a195-5b08-4c9b-b91c-5312624dc72e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.307179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.307099 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kxkv\" (UniqueName: \"kubernetes.io/projected/4864a195-5b08-4c9b-b91c-5312624dc72e-kube-api-access-9kxkv\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.307179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.307136 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4864a195-5b08-4c9b-b91c-5312624dc72e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.314074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.314046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:48.408355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.408322 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4864a195-5b08-4c9b-b91c-5312624dc72e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.408759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.408379 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kxkv\" (UniqueName: \"kubernetes.io/projected/4864a195-5b08-4c9b-b91c-5312624dc72e-kube-api-access-9kxkv\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.408759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.408400 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4864a195-5b08-4c9b-b91c-5312624dc72e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.408982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.408963 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4864a195-5b08-4c9b-b91c-5312624dc72e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.410710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.410692 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4864a195-5b08-4c9b-b91c-5312624dc72e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.433855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.433829 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kxkv\" (UniqueName: \"kubernetes.io/projected/4864a195-5b08-4c9b-b91c-5312624dc72e-kube-api-access-9kxkv\") pod \"kuadrant-console-plugin-66c4fb575d-p6tlb\" (UID: \"4864a195-5b08-4c9b-b91c-5312624dc72e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.470867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.470835 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" Oct 30 07:02:48.767522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.767497 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb"] Oct 30 07:02:48.768838 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:48.768811 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4864a195_5b08_4c9b_b91c_5312624dc72e.slice/crio-ef8eeadd4308adc3a06fa00e75d6e48d7e29a88d856fc05e41fed926bdee99f5 WatchSource:0}: Error finding container ef8eeadd4308adc3a06fa00e75d6e48d7e29a88d856fc05e41fed926bdee99f5: Status 404 returned error can't find the container with id ef8eeadd4308adc3a06fa00e75d6e48d7e29a88d856fc05e41fed926bdee99f5 Oct 30 07:02:48.786556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.786522 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" event={"ID":"01b69050-e4b3-483e-8065-0ef33a849871","Type":"ContainerStarted","Data":"47bd15e5dae0acce969995b02298471cdce4491f1a929bce3c98eda3e3c1a300"} Oct 30 07:02:48.786744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.786627 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:48.787685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.787660 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" event={"ID":"4864a195-5b08-4c9b-b91c-5312624dc72e","Type":"ContainerStarted","Data":"ef8eeadd4308adc3a06fa00e75d6e48d7e29a88d856fc05e41fed926bdee99f5"} Oct 30 07:02:48.788905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.788867 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" event={"ID":"7272d9f0-29bb-4254-8580-850653c638a0","Type":"ContainerStarted","Data":"1fce4483bf61e64169ed863a6cc836b9fa541fee1ac6fd72da860b3a5213fe3e"} Oct 30 07:02:48.832649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:48.832580 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" podStartSLOduration=1.955254417 podStartE2EDuration="4.83256503s" totalImagesPullingTime="2.877310613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:48.832080247 +0000 UTC m=+332.522997685" watchObservedRunningTime="2026-10-30 07:02:48.83256503 +0000 UTC m=+332.523482461" Oct 30 07:02:49.499178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:49.499144 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:49.869985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:49.869941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:51.807453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:51.807409 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" event={"ID":"7272d9f0-29bb-4254-8580-850653c638a0","Type":"ContainerStarted","Data":"e23768c6e81d4ae0c306ae56848ed7e6dac8e4f5965c84fb92efe7f6b9d8c2fc"} Oct 30 07:02:51.807982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:51.807502 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:51.853002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:51.852932 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" podStartSLOduration=1.713914723 podStartE2EDuration="4.85291071s" totalImagesPullingTime="3.138995987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:51.851474708 +0000 UTC m=+335.542392146" watchObservedRunningTime="2026-10-30 07:02:51.85291071 +0000 UTC m=+335.543828148" Oct 30 07:02:53.818139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:53.818047 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" event={"ID":"4864a195-5b08-4c9b-b91c-5312624dc72e","Type":"ContainerStarted","Data":"9b28fc48952a5aa0495086677ac46848860b3fe5dbbdcf59fcf54819587ac3f9"} Oct 30 07:02:53.861746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:53.861686 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-p6tlb" podStartSLOduration=1.065975205 podStartE2EDuration="5.861673159s" totalImagesPullingTime="4.795697954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:53.860976294 +0000 UTC m=+337.551893731" watchObservedRunningTime="2026-10-30 07:02:53.861673159 +0000 UTC m=+337.552590595" Oct 30 07:02:54.943763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:54.943726 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:55.351653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.351604 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v"] Oct 30 07:02:55.355366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.355346 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.375092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.375061 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v"] Oct 30 07:02:55.387954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.387929 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6j7dm\"" Oct 30 07:02:55.474530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.474497 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.474729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.474626 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87j7\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-kube-api-access-b87j7\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.575721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.575681 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.575918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.575846 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87j7\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-kube-api-access-b87j7\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.600983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.600945 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.601156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.601008 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87j7\" (UniqueName: \"kubernetes.io/projected/16931481-2289-4563-bb7b-126731fe0a26-kube-api-access-b87j7\") pod \"cert-manager-cainjector-5c57bbd4fc-lcj8v\" (UID: \"16931481-2289-4563-bb7b-126731fe0a26\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.613829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.613748 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:55.666693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.666657 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" Oct 30 07:02:55.744002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.743941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:55.964750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.964726 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v"] Oct 30 07:02:55.967773 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:55.967747 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16931481_2289_4563_bb7b_126731fe0a26.slice/crio-547b1fbd05a54698d52c084ef79a95891aae7634395774fa16732e4c6a0de363 WatchSource:0}: Error finding container 547b1fbd05a54698d52c084ef79a95891aae7634395774fa16732e4c6a0de363: Status 404 returned error can't find the container with id 547b1fbd05a54698d52c084ef79a95891aae7634395774fa16732e4c6a0de363 Oct 30 07:02:55.988327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:55.988213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:56.432209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.432173 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9"] Oct 30 07:02:56.433648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.433609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:56.436149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.436127 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.455912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.455888 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 07:02:56.459012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.458989 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 07:02:56.465753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.465726 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9"] Oct 30 07:02:56.496459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.496254 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 07:02:56.496459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.496376 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vptpj\"" Oct 30 07:02:56.496459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.496432 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 07:02:56.587057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.586996 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35207e99-083b-4214-bf26-79fabec5c2bf-tmp\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.587239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.587066 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhkt5\" (UniqueName: \"kubernetes.io/projected/35207e99-083b-4214-bf26-79fabec5c2bf-kube-api-access-qhkt5\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.587486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.587460 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35207e99-083b-4214-bf26-79fabec5c2bf-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.658075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.658024 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:56.688557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.688453 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35207e99-083b-4214-bf26-79fabec5c2bf-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.688766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.688716 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35207e99-083b-4214-bf26-79fabec5c2bf-tmp\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.688766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.688760 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhkt5\" (UniqueName: \"kubernetes.io/projected/35207e99-083b-4214-bf26-79fabec5c2bf-kube-api-access-qhkt5\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.691313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.691286 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35207e99-083b-4214-bf26-79fabec5c2bf-tmp\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.691512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.691497 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35207e99-083b-4214-bf26-79fabec5c2bf-tls-certs\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.713964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.713935 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhkt5\" (UniqueName: \"kubernetes.io/projected/35207e99-083b-4214-bf26-79fabec5c2bf-kube-api-access-qhkt5\") pod \"kube-auth-proxy-5cf9d6756f-5mvd9\" (UID: \"35207e99-083b-4214-bf26-79fabec5c2bf\") " pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.750059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.750024 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" Oct 30 07:02:56.805734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.805693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:56.832013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.831981 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" event={"ID":"16931481-2289-4563-bb7b-126731fe0a26","Type":"ContainerStarted","Data":"239d09de55674be8969528e1c1cfb08598a18476c104b65b059b4e9cae121763"} Oct 30 07:02:56.832177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.832024 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" event={"ID":"16931481-2289-4563-bb7b-126731fe0a26","Type":"ContainerStarted","Data":"547b1fbd05a54698d52c084ef79a95891aae7634395774fa16732e4c6a0de363"} Oct 30 07:02:56.878493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.878433 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-lcj8v" podStartSLOduration=1.878419209 podStartE2EDuration="1.878419209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:56.878171549 +0000 UTC m=+340.569088986" watchObservedRunningTime="2026-10-30 07:02:56.878419209 +0000 UTC m=+340.569336646" Oct 30 07:02:56.932829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:56.932790 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:57.049001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.048972 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9"] Oct 30 07:02:57.050728 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:57.050692 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35207e99_083b_4214_bf26_79fabec5c2bf.slice/crio-1b737d34e14f459a7ab8183adcb83858fdf39ee095fe4ef0e5437025f3059259 WatchSource:0}: Error finding container 1b737d34e14f459a7ab8183adcb83858fdf39ee095fe4ef0e5437025f3059259: Status 404 returned error can't find the container with id 1b737d34e14f459a7ab8183adcb83858fdf39ee095fe4ef0e5437025f3059259 Oct 30 07:02:57.052707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.052691 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:02:57.148439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.148399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:57.276697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.276663 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:57.517440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.517366 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:02:57.519892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.519870 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:02:57.547049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.547021 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rqkmz\"" Oct 30 07:02:57.553171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.553142 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:02:57.596472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.596431 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbcw\" (UniqueName: \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\") pod \"authorino-57f7f8659f-ns2dp\" (UID: \"eb2cfd15-4d4d-406e-81db-2873d4e4654a\") " pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:02:57.662565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.662530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:57.697581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.697546 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbcw\" (UniqueName: \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\") pod \"authorino-57f7f8659f-ns2dp\" (UID: \"eb2cfd15-4d4d-406e-81db-2873d4e4654a\") " pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:02:57.743677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.743576 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbcw\" (UniqueName: \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\") pod \"authorino-57f7f8659f-ns2dp\" (UID: \"eb2cfd15-4d4d-406e-81db-2873d4e4654a\") " pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:02:57.815416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.815379 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b4mn5" Oct 30 07:02:57.832699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.832461 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:02:57.834937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.834902 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:57.839888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:57.839859 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" event={"ID":"35207e99-083b-4214-bf26-79fabec5c2bf","Type":"ContainerStarted","Data":"1b737d34e14f459a7ab8183adcb83858fdf39ee095fe4ef0e5437025f3059259"} Oct 30 07:02:58.174152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.174120 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:02:58.175669 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:58.175532 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb2cfd15_4d4d_406e_81db_2873d4e4654a.slice/crio-223a8789ab4c242415be78be37bda41e9ae62b603ffec70315ffcb7f77ce32f8 WatchSource:0}: Error finding container 223a8789ab4c242415be78be37bda41e9ae62b603ffec70315ffcb7f77ce32f8: Status 404 returned error can't find the container with id 223a8789ab4c242415be78be37bda41e9ae62b603ffec70315ffcb7f77ce32f8 Oct 30 07:02:58.290509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.290467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:58.470056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.469975 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:02:58.476238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.476205 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.498907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.498876 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:02:58.505760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.505733 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 07:02:58.534796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.534705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:58.615311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.615266 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13d4950f-8dc2-43d5-8969-842bc081873c-config-file\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.615498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.615321 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8xh\" (UniqueName: \"kubernetes.io/projected/13d4950f-8dc2-43d5-8969-842bc081873c-kube-api-access-5x8xh\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.616299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.616273 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:02:58.716545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.716505 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13d4950f-8dc2-43d5-8969-842bc081873c-config-file\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.716765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.716565 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8xh\" (UniqueName: \"kubernetes.io/projected/13d4950f-8dc2-43d5-8969-842bc081873c-kube-api-access-5x8xh\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.717230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.717202 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/13d4950f-8dc2-43d5-8969-842bc081873c-config-file\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.785536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.785505 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8xh\" (UniqueName: \"kubernetes.io/projected/13d4950f-8dc2-43d5-8969-842bc081873c-kube-api-access-5x8xh\") pod \"limitador-limitador-7c9b87d574-wg9tq\" (UID: \"13d4950f-8dc2-43d5-8969-842bc081873c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.793566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.793336 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:02:58.832528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.832482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:58.854993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:58.853891 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" event={"ID":"eb2cfd15-4d4d-406e-81db-2873d4e4654a","Type":"ContainerStarted","Data":"223a8789ab4c242415be78be37bda41e9ae62b603ffec70315ffcb7f77ce32f8"} Oct 30 07:02:59.105008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.104980 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:02:59.106990 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:02:59.106957 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d4950f_8dc2_43d5_8969_842bc081873c.slice/crio-b7c526f4c5254c4a711179a7f55c49c2478f23924cdc324a7aef8c72b81711b0 WatchSource:0}: Error finding container b7c526f4c5254c4a711179a7f55c49c2478f23924cdc324a7aef8c72b81711b0: Status 404 returned error can't find the container with id b7c526f4c5254c4a711179a7f55c49c2478f23924cdc324a7aef8c72b81711b0 Oct 30 07:02:59.504115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.503857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:59.769700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.769604 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:59.798467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.798436 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-5c2cx" Oct 30 07:02:59.862227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.862189 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" event={"ID":"13d4950f-8dc2-43d5-8969-842bc081873c","Type":"ContainerStarted","Data":"b7c526f4c5254c4a711179a7f55c49c2478f23924cdc324a7aef8c72b81711b0"} Oct 30 07:02:59.864926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.864892 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" event={"ID":"35207e99-083b-4214-bf26-79fabec5c2bf","Type":"ContainerStarted","Data":"1a1a0e0c57921825e18377e82411e7d8fec0c0fb6e0e96fa084fe877bca53d0f"} Oct 30 07:02:59.918706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.914848 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cf9d6756f-5mvd9" podStartSLOduration=1.457730554 podStartE2EDuration="3.914832773s" totalImagesPullingTime="2.457102219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:59.914010639 +0000 UTC m=+343.604928087" watchObservedRunningTime="2026-10-30 07:02:59.914832773 +0000 UTC m=+343.605750209" Oct 30 07:02:59.971477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:02:59.971438 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:00.840948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.840912 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:03:00.870843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.870805 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" event={"ID":"13d4950f-8dc2-43d5-8969-842bc081873c","Type":"ContainerStarted","Data":"5a7df48fa9e8141ad77be1514e332dc47ca4360aa22d65fce6e448ca04518d88"} Oct 30 07:03:00.871002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.870872 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:03:00.872351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.872326 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" event={"ID":"eb2cfd15-4d4d-406e-81db-2873d4e4654a","Type":"ContainerStarted","Data":"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213"} Oct 30 07:03:00.963731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.963680 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" podStartSLOduration=1.7988610889999999 podStartE2EDuration="2.963665856s" totalImagesPullingTime="1.164804767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:00.917224116 +0000 UTC m=+344.608141552" watchObservedRunningTime="2026-10-30 07:03:00.963665856 +0000 UTC m=+344.654583293" Oct 30 07:03:00.964165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:00.964142 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" podStartSLOduration=2.143611977 podStartE2EDuration="3.964135298s" totalImagesPullingTime="1.820523321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:00.96327268 +0000 UTC m=+344.654190121" watchObservedRunningTime="2026-10-30 07:03:00.964135298 +0000 UTC m=+344.655052734" Oct 30 07:03:01.877242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:01.877203 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" podUID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" containerName="authorino" containerID="cri-o://addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213" gracePeriod=30 Oct 30 07:03:02.115947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.115922 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:03:02.252434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.252346 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\" (UniqueName: \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\") pod \"eb2cfd15-4d4d-406e-81db-2873d4e4654a\" (UID: \"eb2cfd15-4d4d-406e-81db-2873d4e4654a\") " Oct 30 07:03:02.254519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.254484 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw" pod "eb2cfd15-4d4d-406e-81db-2873d4e4654a" (UID: "eb2cfd15-4d4d-406e-81db-2873d4e4654a"). InnerVolumeSpecName "kube-api-access-4cbcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:03:02.353344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.353306 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbcw\" (UniqueName: \"kubernetes.io/projected/eb2cfd15-4d4d-406e-81db-2873d4e4654a-kube-api-access-4cbcw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:03:02.883657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.883602 5894 generic.go:356] "Generic (PLEG): container finished" podID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" containerID="addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213" exitCode=0 Oct 30 07:03:02.884054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.883668 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" Oct 30 07:03:02.884054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.883695 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" event={"ID":"eb2cfd15-4d4d-406e-81db-2873d4e4654a","Type":"ContainerDied","Data":"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213"} Oct 30 07:03:02.884054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.883743 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-ns2dp" event={"ID":"eb2cfd15-4d4d-406e-81db-2873d4e4654a","Type":"ContainerDied","Data":"223a8789ab4c242415be78be37bda41e9ae62b603ffec70315ffcb7f77ce32f8"} Oct 30 07:03:02.884054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.883761 5894 scope.go:122] "RemoveContainer" containerID="addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213" Oct 30 07:03:02.891959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.891944 5894 scope.go:122] "RemoveContainer" containerID="addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213" Oct 30 07:03:02.892236 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:02.892216 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213\": container with ID starting with addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213 not found: ID does not exist" containerID="addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213" Oct 30 07:03:02.892305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.892243 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213"} err="failed to get container status \"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213\": rpc error: code = NotFound desc = could not find container \"addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213\": container with ID starting with addd49abba33e3d1831182c8dbf7e83be8f43f21ff4145a73941c8389c0e5213 not found: ID does not exist" Oct 30 07:03:02.942001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.941971 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:03:02.945698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:02.945674 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-ns2dp"] Oct 30 07:03:04.430008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:04.429978 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" path="/var/lib/kubelet/pods/eb2cfd15-4d4d-406e-81db-2873d4e4654a/volumes" Oct 30 07:03:04.718087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:04.717992 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:04.827392 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:04.827350 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:03:04.840179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:04.840147 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:06.433191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.433164 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng"] Oct 30 07:03:06.433584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.433572 5894 cpu_manager.go:414] "containerMap: removing container" podUID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" containerName="authorino" Oct 30 07:03:06.433662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.433586 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" containerName="authorino" Oct 30 07:03:06.433708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.433667 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2cfd15-4d4d-406e-81db-2873d4e4654a" containerName="authorino" Oct 30 07:03:06.436307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.436291 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.455710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.455682 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 07:03:06.455869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.455848 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 07:03:06.470070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.470037 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-bdpx2\"" Oct 30 07:03:06.499012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.498984 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 07:03:06.500274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.500249 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng"] Oct 30 07:03:06.588829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.588794 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.588985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.588860 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.588985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.588936 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnrp8\" (UniqueName: \"kubernetes.io/projected/f0c68c94-cae9-476d-9c8d-560f6c998844-kube-api-access-mnrp8\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.589113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.589001 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c68c94-cae9-476d-9c8d-560f6c998844-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690325 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c68c94-cae9-476d-9c8d-560f6c998844-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690394 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690450 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690491 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnrp8\" (UniqueName: \"kubernetes.io/projected/f0c68c94-cae9-476d-9c8d-560f6c998844-kube-api-access-mnrp8\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690872 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.690939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.690882 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f0c68c94-cae9-476d-9c8d-560f6c998844-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.691310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.691291 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0c68c94-cae9-476d-9c8d-560f6c998844-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.715881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.715849 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnrp8\" (UniqueName: \"kubernetes.io/projected/f0c68c94-cae9-476d-9c8d-560f6c998844-kube-api-access-mnrp8\") pod \"kserve-module-controller-manager-7566cf5fc8-hkjng\" (UID: \"f0c68c94-cae9-476d-9c8d-560f6c998844\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:06.754428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:06.754393 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:07.084670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:07.084645 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng"] Oct 30 07:03:07.086390 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:03:07.086352 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c68c94_cae9_476d_9c8d_560f6c998844.slice/crio-989625cff39e302cbc63dad286b0711c1fac6c880561facd15e526dfa0474ee3 WatchSource:0}: Error finding container 989625cff39e302cbc63dad286b0711c1fac6c880561facd15e526dfa0474ee3: Status 404 returned error can't find the container with id 989625cff39e302cbc63dad286b0711c1fac6c880561facd15e526dfa0474ee3 Oct 30 07:03:07.908049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:07.908015 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" event={"ID":"f0c68c94-cae9-476d-9c8d-560f6c998844","Type":"ContainerStarted","Data":"989625cff39e302cbc63dad286b0711c1fac6c880561facd15e526dfa0474ee3"} Oct 30 07:03:08.914334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:08.914303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" event={"ID":"f0c68c94-cae9-476d-9c8d-560f6c998844","Type":"ContainerStarted","Data":"37cf797890500d841474fafbfcb31ccaa1cb80c8fa03888a2bf3bbdb4f0e6583"} Oct 30 07:03:08.914769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:08.914496 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:08.975565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:08.975510 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" podStartSLOduration=1.576703131 podStartE2EDuration="2.975497344s" totalImagesPullingTime="1.398794213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:08.974627569 +0000 UTC m=+352.665545005" watchObservedRunningTime="2026-10-30 07:03:08.975497344 +0000 UTC m=+352.666414785" Oct 30 07:03:10.469785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:10.469748 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:10.598208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:10.598173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:10.714210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:10.714174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:10.830853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:10.830824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:11.318219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:11.318173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:11.434737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:11.434700 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:11.877849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:11.877823 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-wg9tq" Oct 30 07:03:14.913479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:14.913437 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:15.083313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:15.083273 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:18.327012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:18.326973 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:18.467719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:18.467681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:18.608315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:18.608205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:18.741053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:18.741013 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:19.075781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:19.075744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:24.688253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:24.688213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:25.008753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:25.008664 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:25.124237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:25.124203 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:28.290669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:28.290557 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:28.416397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:28.416362 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:29.725348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:29.725314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:29.851450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:29.851394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:29.920768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:29.920735 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-hkjng" Oct 30 07:03:33.416529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:33.416495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:33.585584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:33.585548 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:34.827125 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:34.827090 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:03:38.106222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.106168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:38.173291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.173198 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:38.180305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.180281 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.202180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.202147 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 07:03:38.213357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.213328 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 07:03:38.213357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.213323 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 07:03:38.213607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.213403 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-q65wk\"" Oct 30 07:03:38.214756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.214732 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:38.229649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.229613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:38.286622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.286587 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.286844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.286721 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgsb7\" (UniqueName: \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.286844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.286818 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.373094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.373045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:38.387732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.387698 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.387904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.387746 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgsb7\" (UniqueName: \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.387904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.387818 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.387904 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:38.387875 5894 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Oct 30 07:03:38.388193 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:38.387976 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert podName:efff5967-f545-4fa4-9829-60fc4c1073d0 nodeName:}" failed. No retries permitted until 2026-10-30 07:03:38.887951247 +0000 UTC m=+382.578868680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert") pod "maas-controller-fcfd895c4-frnrq" (UID: "efff5967-f545-4fa4-9829-60fc4c1073d0") : secret "maas-controller-webhook-cert" not found Oct 30 07:03:38.390618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.390591 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.413591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.413553 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgsb7\" (UniqueName: \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.494876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.494728 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:38.613764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.613728 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:38.699739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.699707 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6454b597dd-mk5z8"] Oct 30 07:03:38.703765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.703742 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.724127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.724098 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6454b597dd-mk5z8"] Oct 30 07:03:38.794653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.794597 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-metrics-certs\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.794851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.794663 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-webhook-cert\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.794851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.794701 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznwv\" (UniqueName: \"kubernetes.io/projected/896ba020-7327-4bc5-88ec-a2292da98092-kube-api-access-tznwv\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.895817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.895783 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.895817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.895828 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-metrics-certs\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.896056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.895866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-webhook-cert\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.896056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.895909 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznwv\" (UniqueName: \"kubernetes.io/projected/896ba020-7327-4bc5-88ec-a2292da98092-kube-api-access-tznwv\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.898974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.898938 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-webhook-cert\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.899285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.899257 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") pod \"maas-controller-fcfd895c4-frnrq\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:38.899409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.899352 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/896ba020-7327-4bc5-88ec-a2292da98092-metrics-certs\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.923091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.923059 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznwv\" (UniqueName: \"kubernetes.io/projected/896ba020-7327-4bc5-88ec-a2292da98092-kube-api-access-tznwv\") pod \"maas-controller-6454b597dd-mk5z8\" (UID: \"896ba020-7327-4bc5-88ec-a2292da98092\") " pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:38.963784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:38.963746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:39.017191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.017154 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:39.093314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.093284 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:39.097446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.097408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:39.220108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.220076 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:39.317547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.317496 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6454b597dd-mk5z8"] Oct 30 07:03:39.318756 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:03:39.318729 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896ba020_7327_4bc5_88ec_a2292da98092.slice/crio-088fbf9c148178b6af76895c314756bb5fa76a4cd5f4707a281a20431d95ba78 WatchSource:0}: Error finding container 088fbf9c148178b6af76895c314756bb5fa76a4cd5f4707a281a20431d95ba78: Status 404 returned error can't find the container with id 088fbf9c148178b6af76895c314756bb5fa76a4cd5f4707a281a20431d95ba78 Oct 30 07:03:39.396438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:39.396409 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:39.398197 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:03:39.398164 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefff5967_f545_4fa4_9829_60fc4c1073d0.slice/crio-854a7053344ea7f1d594a9f53e28f8b8003a553eb961d10991dd00384c41b1e3 WatchSource:0}: Error finding container 854a7053344ea7f1d594a9f53e28f8b8003a553eb961d10991dd00384c41b1e3: Status 404 returned error can't find the container with id 854a7053344ea7f1d594a9f53e28f8b8003a553eb961d10991dd00384c41b1e3 Oct 30 07:03:40.059120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:40.059043 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6454b597dd-mk5z8" event={"ID":"896ba020-7327-4bc5-88ec-a2292da98092","Type":"ContainerStarted","Data":"088fbf9c148178b6af76895c314756bb5fa76a4cd5f4707a281a20431d95ba78"} Oct 30 07:03:40.062154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:40.062084 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-frnrq" event={"ID":"efff5967-f545-4fa4-9829-60fc4c1073d0","Type":"ContainerStarted","Data":"854a7053344ea7f1d594a9f53e28f8b8003a553eb961d10991dd00384c41b1e3"} Oct 30 07:03:42.643067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:42.643032 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:42.758276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:42.758183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:43.080584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.080540 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6454b597dd-mk5z8" event={"ID":"896ba020-7327-4bc5-88ec-a2292da98092","Type":"ContainerStarted","Data":"142a4b9f63979dcd0b9fec1d58586fdc2de010e0b5ccf2711c9cbefe7358d200"} Oct 30 07:03:43.080817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.080664 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:43.082345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.082318 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-frnrq" event={"ID":"efff5967-f545-4fa4-9829-60fc4c1073d0","Type":"ContainerStarted","Data":"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c"} Oct 30 07:03:43.082498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.082437 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:43.128018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.127962 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6454b597dd-mk5z8" podStartSLOduration=2.048770215 podStartE2EDuration="5.127945532s" totalImagesPullingTime="3.079175317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:43.126701042 +0000 UTC m=+386.817618480" watchObservedRunningTime="2026-10-30 07:03:43.127945532 +0000 UTC m=+386.818862966" Oct 30 07:03:43.174343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:43.174279 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-fcfd895c4-frnrq" podStartSLOduration=2.133286086 podStartE2EDuration="5.174256853s" totalImagesPullingTime="3.040970767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:43.172853265 +0000 UTC m=+386.863770705" watchObservedRunningTime="2026-10-30 07:03:43.174256853 +0000 UTC m=+386.865174290" Oct 30 07:03:44.146864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.146832 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp"] Oct 30 07:03:44.153425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.153399 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.180177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.175089 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp"] Oct 30 07:03:44.185357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.185320 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 07:03:44.185527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.185374 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 07:03:44.185744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.185712 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jsgl6\"" Oct 30 07:03:44.187411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.187386 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 07:03:44.238489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.238444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:44.264895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.264864 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzf7\" (UniqueName: \"kubernetes.io/projected/4c1f0382-bd87-4917-85bc-0054889e2c44-kube-api-access-pjzf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.264895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.264897 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.265142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.264932 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.265252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.265228 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c1f0382-bd87-4917-85bc-0054889e2c44-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.353532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.353495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:44.366848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.366812 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c1f0382-bd87-4917-85bc-0054889e2c44-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.367023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.366868 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzf7\" (UniqueName: \"kubernetes.io/projected/4c1f0382-bd87-4917-85bc-0054889e2c44-kube-api-access-pjzf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.367023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.366901 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.367023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.366933 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.367750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.367727 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.367905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.367885 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c1f0382-bd87-4917-85bc-0054889e2c44-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.369626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.369602 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c1f0382-bd87-4917-85bc-0054889e2c44-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.392377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.392333 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzf7\" (UniqueName: \"kubernetes.io/projected/4c1f0382-bd87-4917-85bc-0054889e2c44-kube-api-access-pjzf7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-zmvtp\" (UID: \"4c1f0382-bd87-4917-85bc-0054889e2c44\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.466306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.466212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:03:44.466988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.466964 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:44.773744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:44.773714 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp"] Oct 30 07:03:44.774734 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:03:44.774706 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1f0382_bd87_4917_85bc_0054889e2c44.slice/crio-0f02aaae74ad0ed3e05d1cc338904514fe9e7cbb7443d675b78f79de29fe23c2 WatchSource:0}: Error finding container 0f02aaae74ad0ed3e05d1cc338904514fe9e7cbb7443d675b78f79de29fe23c2: Status 404 returned error can't find the container with id 0f02aaae74ad0ed3e05d1cc338904514fe9e7cbb7443d675b78f79de29fe23c2 Oct 30 07:03:45.093784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:45.093744 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" event={"ID":"4c1f0382-bd87-4917-85bc-0054889e2c44","Type":"ContainerStarted","Data":"0f02aaae74ad0ed3e05d1cc338904514fe9e7cbb7443d675b78f79de29fe23c2"} Oct 30 07:03:54.089310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.089277 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:54.089942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.089434 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:03:54.167101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.167065 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:54.167324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.167278 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-fcfd895c4-frnrq" podUID="efff5967-f545-4fa4-9829-60fc4c1073d0" containerName="manager" containerID="cri-o://27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c" gracePeriod=30 Oct 30 07:03:54.451937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.451901 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:54.467426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.466606 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\") pod \"efff5967-f545-4fa4-9829-60fc4c1073d0\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " Oct 30 07:03:54.467426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.466750 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") pod \"efff5967-f545-4fa4-9829-60fc4c1073d0\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " Oct 30 07:03:54.467426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.466842 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\" (UniqueName: \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\") pod \"efff5967-f545-4fa4-9829-60fc4c1073d0\" (UID: \"efff5967-f545-4fa4-9829-60fc4c1073d0\") " Oct 30 07:03:54.469920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.469885 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7" pod "efff5967-f545-4fa4-9829-60fc4c1073d0" (UID: "efff5967-f545-4fa4-9829-60fc4c1073d0"). InnerVolumeSpecName "kube-api-access-tgsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:03:54.469920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.469900 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs" pod "efff5967-f545-4fa4-9829-60fc4c1073d0" (UID: "efff5967-f545-4fa4-9829-60fc4c1073d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:03:54.470117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.470094 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert" pod "efff5967-f545-4fa4-9829-60fc4c1073d0" (UID: "efff5967-f545-4fa4-9829-60fc4c1073d0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:03:54.568110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.568070 5894 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-webhook-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:03:54.568110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.568114 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgsb7\" (UniqueName: \"kubernetes.io/projected/efff5967-f545-4fa4-9829-60fc4c1073d0-kube-api-access-tgsb7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:03:54.568361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:54.568132 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efff5967-f545-4fa4-9829-60fc4c1073d0-metrics-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:03:55.151628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.151592 5894 generic.go:356] "Generic (PLEG): container finished" podID="efff5967-f545-4fa4-9829-60fc4c1073d0" containerID="27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c" exitCode=0 Oct 30 07:03:55.152168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.151671 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-frnrq" event={"ID":"efff5967-f545-4fa4-9829-60fc4c1073d0","Type":"ContainerDied","Data":"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c"} Oct 30 07:03:55.152168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.151745 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fcfd895c4-frnrq" event={"ID":"efff5967-f545-4fa4-9829-60fc4c1073d0","Type":"ContainerDied","Data":"854a7053344ea7f1d594a9f53e28f8b8003a553eb961d10991dd00384c41b1e3"} Oct 30 07:03:55.152168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.151684 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fcfd895c4-frnrq" Oct 30 07:03:55.152168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.151779 5894 scope.go:122] "RemoveContainer" containerID="27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c" Oct 30 07:03:55.163789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.163765 5894 scope.go:122] "RemoveContainer" containerID="27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c" Oct 30 07:03:55.164141 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:03:55.164115 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c\": container with ID starting with 27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c not found: ID does not exist" containerID="27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c" Oct 30 07:03:55.164227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.164154 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c"} err="failed to get container status \"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c\": rpc error: code = NotFound desc = could not find container \"27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c\": container with ID starting with 27fd439acc79f727dffb7d123bcc1aa61fec1038ec13bd30b25ef97b70a0f25c not found: ID does not exist" Oct 30 07:03:55.208997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.208957 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:55.212904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:55.212877 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fcfd895c4-frnrq"] Oct 30 07:03:56.158014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:56.157976 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" event={"ID":"4c1f0382-bd87-4917-85bc-0054889e2c44","Type":"ContainerStarted","Data":"1f69da769cc05fdf704405c355cbd61fab177e39b71ca4ca31971a6030f25fba"} Oct 30 07:03:56.158476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:56.158042 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:03:56.204212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:56.204153 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" podStartSLOduration=1.346115772 podStartE2EDuration="12.204139576s" totalImagesPullingTime="10.858023804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:03:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:03:56.203340992 +0000 UTC m=+399.894258429" watchObservedRunningTime="2026-10-30 07:03:56.204139576 +0000 UTC m=+399.895057013" Oct 30 07:03:56.429320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:03:56.429245 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efff5967-f545-4fa4-9829-60fc4c1073d0" path="/var/lib/kubelet/pods/efff5967-f545-4fa4-9829-60fc4c1073d0/volumes" Oct 30 07:04:04.760724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:04.760675 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:04.828858 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:04:04.828819 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:04:04.888384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:04.888351 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.007561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.007525 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.024166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.024102 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jg24c"] Oct 30 07:04:05.024544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.024530 5894 cpu_manager.go:414] "containerMap: removing container" podUID="efff5967-f545-4fa4-9829-60fc4c1073d0" containerName="manager" Oct 30 07:04:05.024591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.024545 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efff5967-f545-4fa4-9829-60fc4c1073d0" containerName="manager" Oct 30 07:04:05.024628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.024610 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="efff5967-f545-4fa4-9829-60fc4c1073d0" containerName="manager" Oct 30 07:04:05.032175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.032150 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.048970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.048941 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jg24c"] Oct 30 07:04:05.061948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.061921 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 07:04:05.067251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.067228 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qc2l2\"" Oct 30 07:04:05.079329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.078277 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:04:05.085593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.085556 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.101506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.101476 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:04:05.112452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.112417 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nlccq"] Oct 30 07:04:05.118057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.118035 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.142669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.142622 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nlccq"] Oct 30 07:04:05.156560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.156528 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.160968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.160951 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 07:04:05.161464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.161440 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 07:04:05.161464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.161454 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lpdg6\"" Oct 30 07:04:05.161744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.161727 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-s4mhp\"" Oct 30 07:04:05.173205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173177 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05999c41-8b41-470c-8762-87620709e24e-cert\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.173335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173222 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-tls-certs\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.173335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173323 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b792b\" (UniqueName: \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.173452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173382 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz6fn\" (UniqueName: \"kubernetes.io/projected/05999c41-8b41-470c-8762-87620709e24e-kube-api-access-zz6fn\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.173452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173416 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.173452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.173443 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257v9\" (UniqueName: \"kubernetes.io/projected/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-kube-api-access-257v9\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.231797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.231763 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-gqtn9"] Oct 30 07:04:05.235265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.235247 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.267344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.267313 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-cq2lb\"" Oct 30 07:04:05.267344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.267313 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 07:04:05.267565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.267434 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 07:04:05.271892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.271867 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-gqtn9"] Oct 30 07:04:05.274826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.274762 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05999c41-8b41-470c-8762-87620709e24e-cert\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.274826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.274809 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-tls-certs\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.275013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.274845 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b792b\" (UniqueName: \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.275013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.274874 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz6fn\" (UniqueName: \"kubernetes.io/projected/05999c41-8b41-470c-8762-87620709e24e-kube-api-access-zz6fn\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.275013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.274891 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.275149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.275035 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257v9\" (UniqueName: \"kubernetes.io/projected/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-kube-api-access-257v9\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.277574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.277545 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-tls-certs\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.277710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.277581 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.277710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.277588 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05999c41-8b41-470c-8762-87620709e24e-cert\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.300790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.300761 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz6fn\" (UniqueName: \"kubernetes.io/projected/05999c41-8b41-470c-8762-87620709e24e-kube-api-access-zz6fn\") pod \"kserve-controller-manager-776cc66d47-jg24c\" (UID: \"05999c41-8b41-470c-8762-87620709e24e\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.300934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.300762 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257v9\" (UniqueName: \"kubernetes.io/projected/6eb85c1d-f139-4bb3-a299-e9a7dae8a683-kube-api-access-257v9\") pod \"model-serving-api-75bd89fdfc-nlccq\" (UID: \"6eb85c1d-f139-4bb3-a299-e9a7dae8a683\") " pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.300976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.300950 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b792b\" (UniqueName: \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\") pod \"llmisvc-controller-manager-869d5dcc5d-dhbvz\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.326772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.326715 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.346791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.346754 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:05.376486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.376438 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-metrics-certs\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.376698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.376596 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-cert\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.376698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.376692 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mds7\" (UniqueName: \"kubernetes.io/projected/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-kube-api-access-8mds7\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.402417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.402382 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:05.432943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.431949 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:05.450306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.450248 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.477848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.477806 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-cert\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.478023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.477895 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mds7\" (UniqueName: \"kubernetes.io/projected/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-kube-api-access-8mds7\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.478023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.477956 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-metrics-certs\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.481251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.481211 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-metrics-certs\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.483366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.483321 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-cert\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.508668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.504812 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mds7\" (UniqueName: \"kubernetes.io/projected/b4518ef6-492e-4c5b-8e5c-6bc7669a294d-kube-api-access-8mds7\") pod \"odh-model-controller-769875db56-gqtn9\" (UID: \"b4518ef6-492e-4c5b-8e5c-6bc7669a294d\") " pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.551242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.548278 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:05.591189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.591146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:05.650624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.650593 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jg24c"] Oct 30 07:04:05.652846 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:04:05.652815 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05999c41_8b41_470c_8762_87620709e24e.slice/crio-84177e58e4b804acc543343c1e782023e0c3c0df0e2d0554b4c0c37db1e7cd3f WatchSource:0}: Error finding container 84177e58e4b804acc543343c1e782023e0c3c0df0e2d0554b4c0c37db1e7cd3f: Status 404 returned error can't find the container with id 84177e58e4b804acc543343c1e782023e0c3c0df0e2d0554b4c0c37db1e7cd3f Oct 30 07:04:05.736320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.736290 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:04:05.737077 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:04:05.737051 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf2dec874_20db_49bc_b646_3304638a30ab.slice/crio-ea6888ce457c7e55062b21cacfadf18cd1452beb1765bbd26328161b9a01842d WatchSource:0}: Error finding container ea6888ce457c7e55062b21cacfadf18cd1452beb1765bbd26328161b9a01842d: Status 404 returned error can't find the container with id ea6888ce457c7e55062b21cacfadf18cd1452beb1765bbd26328161b9a01842d Oct 30 07:04:05.776386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.776317 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-nlccq"] Oct 30 07:04:05.777487 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:04:05.777463 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb85c1d_f139_4bb3_a299_e9a7dae8a683.slice/crio-95b888d6bbb144ba637e0786a48a7cf6bb22642707015617c6a2c3e3383f4f11 WatchSource:0}: Error finding container 95b888d6bbb144ba637e0786a48a7cf6bb22642707015617c6a2c3e3383f4f11: Status 404 returned error can't find the container with id 95b888d6bbb144ba637e0786a48a7cf6bb22642707015617c6a2c3e3383f4f11 Oct 30 07:04:05.855677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:05.855652 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-gqtn9"] Oct 30 07:04:05.856570 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:04:05.856540 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4518ef6_492e_4c5b_8e5c_6bc7669a294d.slice/crio-45adc9083778fdc545204e03fba109f250f6fa42edd704caeeafe35793f65eeb WatchSource:0}: Error finding container 45adc9083778fdc545204e03fba109f250f6fa42edd704caeeafe35793f65eeb: Status 404 returned error can't find the container with id 45adc9083778fdc545204e03fba109f250f6fa42edd704caeeafe35793f65eeb Oct 30 07:04:06.208197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:06.208162 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-gqtn9" event={"ID":"b4518ef6-492e-4c5b-8e5c-6bc7669a294d","Type":"ContainerStarted","Data":"45adc9083778fdc545204e03fba109f250f6fa42edd704caeeafe35793f65eeb"} Oct 30 07:04:06.209662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:06.209618 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" event={"ID":"6eb85c1d-f139-4bb3-a299-e9a7dae8a683","Type":"ContainerStarted","Data":"95b888d6bbb144ba637e0786a48a7cf6bb22642707015617c6a2c3e3383f4f11"} Oct 30 07:04:06.211144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:06.211115 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" event={"ID":"05999c41-8b41-470c-8762-87620709e24e","Type":"ContainerStarted","Data":"84177e58e4b804acc543343c1e782023e0c3c0df0e2d0554b4c0c37db1e7cd3f"} Oct 30 07:04:06.212562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:06.212534 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" event={"ID":"f2dec874-20db-49bc-b646-3304638a30ab","Type":"ContainerStarted","Data":"ea6888ce457c7e55062b21cacfadf18cd1452beb1765bbd26328161b9a01842d"} Oct 30 07:04:07.171955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.171915 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-zmvtp" Oct 30 07:04:07.438345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.438246 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:07.564353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.564295 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:07.680475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.680439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:07.797054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.797020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:07.917066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:07.917004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:09.243574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.243481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" event={"ID":"05999c41-8b41-470c-8762-87620709e24e","Type":"ContainerStarted","Data":"ae5d1e1d675c6d927f4aaede49fd33e8dde3d40387143f53cb5c3a4b0bcdedda"} Oct 30 07:04:09.243574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.243571 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:04:09.245619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.245581 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" event={"ID":"f2dec874-20db-49bc-b646-3304638a30ab","Type":"ContainerStarted","Data":"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955"} Oct 30 07:04:09.245788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.245708 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:09.247412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.247381 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" event={"ID":"6eb85c1d-f139-4bb3-a299-e9a7dae8a683","Type":"ContainerStarted","Data":"0864ad090ed4ae4309dab9cc0fa3fbab580a7397eb7e6455144022a409bc8975"} Oct 30 07:04:09.247944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.247924 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:09.309461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.309400 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" podStartSLOduration=1.2235764 podStartE2EDuration="4.309386616s" totalImagesPullingTime="3.085810216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:09.308755453 +0000 UTC m=+412.999672890" watchObservedRunningTime="2026-10-30 07:04:09.309386616 +0000 UTC m=+413.000304054" Oct 30 07:04:09.369179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.369126 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" podStartSLOduration=1.481146128 podStartE2EDuration="4.369111187s" totalImagesPullingTime="2.887965059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:09.367899054 +0000 UTC m=+413.058816488" watchObservedRunningTime="2026-10-30 07:04:09.369111187 +0000 UTC m=+413.060028664" Oct 30 07:04:09.415877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:09.415829 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" podStartSLOduration=1.669688166 podStartE2EDuration="4.415813555s" totalImagesPullingTime="2.746125389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:09.414360362 +0000 UTC m=+413.105277801" watchObservedRunningTime="2026-10-30 07:04:09.415813555 +0000 UTC m=+413.106730992" Oct 30 07:04:10.253742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:10.253706 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-gqtn9" event={"ID":"b4518ef6-492e-4c5b-8e5c-6bc7669a294d","Type":"ContainerStarted","Data":"381db36bae1054d43067a224a7fd5ce786c616f8833f01a5aaadecf57c67e53f"} Oct 30 07:04:10.254214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:10.254007 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:10.300757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:10.300699 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-gqtn9" podStartSLOduration=1.786838564 podStartE2EDuration="5.300684664s" totalImagesPullingTime="3.5138461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:04:10.300018902 +0000 UTC m=+413.990936338" watchObservedRunningTime="2026-10-30 07:04:10.300684664 +0000 UTC m=+413.991602097" Oct 30 07:04:12.850227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:12.850190 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:12.974915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:12.974881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:14.414306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:14.414273 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:14.535604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:14.535555 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:14.654731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:14.654684 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:17.424980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:17.424937 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:04:18.704884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:18.704852 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:21.261741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:21.261710 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-gqtn9" Oct 30 07:04:21.263674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:21.263649 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-nlccq" Oct 30 07:04:24.344819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:24.344776 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:24.724456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:24.724360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:24.859410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:24.859372 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:04:34.826649 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:04:34.826590 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:04:40.256599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:40.256563 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:04:40.259759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:04:40.259730 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jg24c" Oct 30 07:05:04.828238 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:05:04.828160 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:05:08.616246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.616210 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9"] Oct 30 07:05:08.620118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.620092 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.639483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.639446 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9"] Oct 30 07:05:08.713064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.713029 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b51d8f-fd83-4c55-8b43-e830d2739434-cert\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.713266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.713099 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcq8\" (UniqueName: \"kubernetes.io/projected/85b51d8f-fd83-4c55-8b43-e830d2739434-kube-api-access-mhcq8\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.814628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.814585 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcq8\" (UniqueName: \"kubernetes.io/projected/85b51d8f-fd83-4c55-8b43-e830d2739434-kube-api-access-mhcq8\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.814867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.814714 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b51d8f-fd83-4c55-8b43-e830d2739434-cert\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.817135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.817114 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/85b51d8f-fd83-4c55-8b43-e830d2739434-cert\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.840450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.840414 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcq8\" (UniqueName: \"kubernetes.io/projected/85b51d8f-fd83-4c55-8b43-e830d2739434-kube-api-access-mhcq8\") pod \"llmisvc-controller-manager-6466c554d8-2dkx9\" (UID: \"85b51d8f-fd83-4c55-8b43-e830d2739434\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:08.933007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:08.932919 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:09.231627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:09.231599 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9"] Oct 30 07:05:09.232492 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:05:09.232468 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b51d8f_fd83_4c55_8b43_e830d2739434.slice/crio-179815b385ee0e3ea49d026cdc8ac7184523f9b304da428b3073bc879ccd1947 WatchSource:0}: Error finding container 179815b385ee0e3ea49d026cdc8ac7184523f9b304da428b3073bc879ccd1947: Status 404 returned error can't find the container with id 179815b385ee0e3ea49d026cdc8ac7184523f9b304da428b3073bc879ccd1947 Oct 30 07:05:09.552468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:09.552432 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" event={"ID":"85b51d8f-fd83-4c55-8b43-e830d2739434","Type":"ContainerStarted","Data":"179815b385ee0e3ea49d026cdc8ac7184523f9b304da428b3073bc879ccd1947"} Oct 30 07:05:10.558865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:10.558830 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" event={"ID":"85b51d8f-fd83-4c55-8b43-e830d2739434","Type":"ContainerStarted","Data":"deed5a36988e3f71263a2d80b59d7018bf235612e17e697d7387650ff25ad665"} Oct 30 07:05:10.559340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:10.558916 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:10.603777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:10.603712 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" podStartSLOduration=2.258235815 podStartE2EDuration="2.603695801s" totalImagesPullingTime="345.459986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:05:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:05:10.602894162 +0000 UTC m=+474.293811580" watchObservedRunningTime="2026-10-30 07:05:10.603695801 +0000 UTC m=+474.294613238" Oct 30 07:05:24.963185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:24.963152 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:05:34.826522 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:05:34.826474 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:05:41.566230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.566199 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-2dkx9" Oct 30 07:05:41.648500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.648449 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:05:41.648906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.648787 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" podUID="f2dec874-20db-49bc-b646-3304638a30ab" containerName="manager" containerID="cri-o://2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955" gracePeriod=30 Oct 30 07:05:41.903426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.903398 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:05:41.942886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.942854 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\" (UniqueName: \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\") pod \"f2dec874-20db-49bc-b646-3304638a30ab\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " Oct 30 07:05:41.943049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.942905 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\" (UniqueName: \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\") pod \"f2dec874-20db-49bc-b646-3304638a30ab\" (UID: \"f2dec874-20db-49bc-b646-3304638a30ab\") " Oct 30 07:05:41.944980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.944946 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b" pod "f2dec874-20db-49bc-b646-3304638a30ab" (UID: "f2dec874-20db-49bc-b646-3304638a30ab"). InnerVolumeSpecName "kube-api-access-b792b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:05:41.945109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:41.945025 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert" pod "f2dec874-20db-49bc-b646-3304638a30ab" (UID: "f2dec874-20db-49bc-b646-3304638a30ab"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:05:42.043742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.043705 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b792b\" (UniqueName: \"kubernetes.io/projected/f2dec874-20db-49bc-b646-3304638a30ab-kube-api-access-b792b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:05:42.043742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.043737 5894 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2dec874-20db-49bc-b646-3304638a30ab-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:05:42.711743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.711654 5894 generic.go:356] "Generic (PLEG): container finished" podID="f2dec874-20db-49bc-b646-3304638a30ab" containerID="2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955" exitCode=0 Oct 30 07:05:42.711743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.711704 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" Oct 30 07:05:42.711743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.711719 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" event={"ID":"f2dec874-20db-49bc-b646-3304638a30ab","Type":"ContainerDied","Data":"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955"} Oct 30 07:05:42.712309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.711780 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz" event={"ID":"f2dec874-20db-49bc-b646-3304638a30ab","Type":"ContainerDied","Data":"ea6888ce457c7e55062b21cacfadf18cd1452beb1765bbd26328161b9a01842d"} Oct 30 07:05:42.712309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.711807 5894 scope.go:122] "RemoveContainer" containerID="2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955" Oct 30 07:05:42.720606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.720585 5894 scope.go:122] "RemoveContainer" containerID="2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955" Oct 30 07:05:42.720905 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:05:42.720885 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955\": container with ID starting with 2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955 not found: ID does not exist" containerID="2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955" Oct 30 07:05:42.720976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.720917 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955"} err="failed to get container status \"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955\": rpc error: code = NotFound desc = could not find container \"2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955\": container with ID starting with 2c1fe18a44ee17ca2c25bf44c100837e524be7f501135c53309432ffa8fd6955 not found: ID does not exist" Oct 30 07:05:42.771157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.771122 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:05:42.785475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:42.785442 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-dhbvz"] Oct 30 07:05:43.095671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:43.095625 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:05:43.217033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:43.217001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:05:44.430410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:44.430374 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2dec874-20db-49bc-b646-3304638a30ab" path="/var/lib/kubelet/pods/f2dec874-20db-49bc-b646-3304638a30ab/volumes" Oct 30 07:05:47.378086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:47.378053 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:05:47.506736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:05:47.506695 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:04.827382 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:06:04.827349 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:06:11.557083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:11.557040 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:11.860823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:11.860790 5894 generic.go:356] "Generic (PLEG): container finished" podID="896ba020-7327-4bc5-88ec-a2292da98092" containerID="142a4b9f63979dcd0b9fec1d58586fdc2de010e0b5ccf2711c9cbefe7358d200" exitCode=1 Oct 30 07:06:11.860991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:11.860875 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6454b597dd-mk5z8" event={"ID":"896ba020-7327-4bc5-88ec-a2292da98092","Type":"ContainerDied","Data":"142a4b9f63979dcd0b9fec1d58586fdc2de010e0b5ccf2711c9cbefe7358d200"} Oct 30 07:06:11.861382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:11.861364 5894 scope.go:122] "RemoveContainer" containerID="142a4b9f63979dcd0b9fec1d58586fdc2de010e0b5ccf2711c9cbefe7358d200" Oct 30 07:06:11.907618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:11.907575 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:12.452401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:12.452368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:12.867884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:12.867852 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6454b597dd-mk5z8" event={"ID":"896ba020-7327-4bc5-88ec-a2292da98092","Type":"ContainerStarted","Data":"bf6f4a22f7847d44c2d5dc69e692249421db7673d91983cb90da6757eedbdb85"} Oct 30 07:06:12.868303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:12.868067 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:06:13.739849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:13.739808 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:13.864015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:13.863979 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:23.876347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:23.876317 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6454b597dd-mk5z8" Oct 30 07:06:25.088575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:25.088530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:29.491275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:29.491181 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:30.389224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:30.389186 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:06:32.962307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:32.962266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:34.060982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:34.060942 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:06:34.706569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:34.706518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:34.828606 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:06:34.828573 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:06:37.030741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:37.030699 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:37.997907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:37.997875 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:06:40.333398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:40.333366 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:41.367644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:41.367606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:41.594439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:41.594402 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:06:43.049476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:43.049439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:46.191594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:46.191553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:06:48.220760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:48.220721 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:06:55.728984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.728949 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:06:55.729582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.729416 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f2dec874-20db-49bc-b646-3304638a30ab" containerName="manager" Oct 30 07:06:55.729582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.729427 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dec874-20db-49bc-b646-3304638a30ab" containerName="manager" Oct 30 07:06:55.729582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.729516 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dec874-20db-49bc-b646-3304638a30ab" containerName="manager" Oct 30 07:06:55.732937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.732917 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.757245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.757203 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 07:06:55.767501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.767461 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:06:55.837940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.837902 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.837961 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838049 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkk2f\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838066 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j9rfz\"" Oct 30 07:06:55.838139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838082 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838133 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838203 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838238 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838270 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.838523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.838401 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939541 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939581 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939722 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939782 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939859 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.939925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939897 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939953 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkk2f\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939983 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.939997 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.940032 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.940519 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.940577 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.940605 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.940810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.940696 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.942121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.942101 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.942368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.942346 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.965315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.965285 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:55.965501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:55.965476 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkk2f\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\") pod \"maas-default-gateway-openshift-default-6dcf757899-cvdjt\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:56.047895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:56.047857 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:06:56.368726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:56.368703 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:06:56.370002 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:06:56.369975 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9facc6_315b_4f14_95d8_e4e3de5928d2.slice/crio-8932f5662e18b0e45ad5b891e09c80045e6debdd862fb89415e123f2811a0f86 WatchSource:0}: Error finding container 8932f5662e18b0e45ad5b891e09c80045e6debdd862fb89415e123f2811a0f86: Status 404 returned error can't find the container with id 8932f5662e18b0e45ad5b891e09c80045e6debdd862fb89415e123f2811a0f86 Oct 30 07:06:57.088349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:57.088305 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" event={"ID":"2b9facc6-315b-4f14-95d8-e4e3de5928d2","Type":"ContainerStarted","Data":"8932f5662e18b0e45ad5b891e09c80045e6debdd862fb89415e123f2811a0f86"} Oct 30 07:06:59.148594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:59.148558 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:06:59.148897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:59.148645 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:06:59.148897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:06:59.148677 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:00.107757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:00.107721 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" event={"ID":"2b9facc6-315b-4f14-95d8-e4e3de5928d2","Type":"ContainerStarted","Data":"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671"} Oct 30 07:07:00.194672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:00.193218 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podStartSLOduration=2.417479433 podStartE2EDuration="5.193192388s" totalImagesPullingTime="2.775712955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:00.173943463 +0000 UTC m=+583.864860901" watchObservedRunningTime="2026-10-30 07:07:00.193192388 +0000 UTC m=+583.884109827" Oct 30 07:07:01.048511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:01.048471 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:07:01.050097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:01.050069 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:01.050249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:01.050127 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:02.048789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:02.048751 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:02.049183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:02.048818 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:02.909461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:02.909417 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:03.035238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:03.035196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:03.048248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:03.048212 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:03.048422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:03.048281 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:04.048922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:04.048891 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:04.049303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:04.048945 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:04.553828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:04.553790 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:04.691728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:04.691679 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:04.834197 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:07:04.834098 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:07:05.048306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:05.048266 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:05.048477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:05.048326 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:06.048449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:06.048403 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-cvdjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 07:07:06.048894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:06.048467 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 07:07:07.427233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:07.427197 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:07:07.427701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:07.427577 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:07:07.428300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:07.428277 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:07:10.780311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.780276 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:07:10.784224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.784206 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810087 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810126 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810199 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kbv\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810266 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810307 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810341 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810503 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.810552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.810527 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.823141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.823109 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:07:10.911414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911365 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911415 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911477 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911507 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96kbv\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911581 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911683 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911715 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.911789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911742 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.912138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911859 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.912138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.911893 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.912231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.912128 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.912269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.912224 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.912315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.912294 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.914055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.914036 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.914613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.914594 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.937158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.937122 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:10.937279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:10.937255 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kbv\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\") pod \"maas-default-gateway-openshift-default-6dcf757899-fjz4n\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:11.099759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:11.099668 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:11.415658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:11.415614 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:07:11.416593 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:07:11.416571 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12eae9a0_97ce_438f_8dfd_45b6a83319a8.slice/crio-e665afb173e8e764e98639c46738eb852447575756d045d9e87917346ce5240e WatchSource:0}: Error finding container e665afb173e8e764e98639c46738eb852447575756d045d9e87917346ce5240e: Status 404 returned error can't find the container with id e665afb173e8e764e98639c46738eb852447575756d045d9e87917346ce5240e Oct 30 07:07:11.419527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:11.419494 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:11.419616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:11.419563 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:11.419616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:11.419592 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:12.169936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:12.169897 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" event={"ID":"12eae9a0-97ce-438f-8dfd-45b6a83319a8","Type":"ContainerStarted","Data":"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39"} Oct 30 07:07:12.170375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:12.169959 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" event={"ID":"12eae9a0-97ce-438f-8dfd-45b6a83319a8","Type":"ContainerStarted","Data":"e665afb173e8e764e98639c46738eb852447575756d045d9e87917346ce5240e"} Oct 30 07:07:12.237977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:12.237928 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podStartSLOduration=2.237911373 podStartE2EDuration="2.237911373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:12.237572642 +0000 UTC m=+595.928490079" watchObservedRunningTime="2026-10-30 07:07:12.237911373 +0000 UTC m=+595.928828809" Oct 30 07:07:13.100810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:13.100777 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:13.102087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:13.102055 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:13.102235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:13.102113 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:14.101123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:14.101086 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:14.101650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:14.101150 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:15.100746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:15.100707 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:15.100938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:15.100784 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:16.100704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:16.100646 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:16.101114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:16.100750 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:17.100220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:17.100182 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:17.100387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:17.100247 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:18.100721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:18.100666 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-fjz4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 07:07:18.101224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:18.100725 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 07:07:19.678100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:19.678065 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:19.678618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:19.678526 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:19.679360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:19.679331 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:07:25.199001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.198962 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:25.811795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.811750 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn"] Oct 30 07:07:25.815849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.815830 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.838098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.838068 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn"] Oct 30 07:07:25.985780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.985740 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.985994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.985797 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.985994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.985958 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986014 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986071 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986169 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986203 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986219 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kvt4\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-kube-api-access-4kvt4\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:25.986311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:25.986248 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087052 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087105 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kvt4\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-kube-api-access-4kvt4\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087161 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087241 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087283 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087324 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087362 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087388 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087673 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087851 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087874 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.087982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.087959 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.088052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.088027 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.090198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.090168 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.090302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.090289 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.113141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.113108 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kvt4\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-kube-api-access-4kvt4\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.113260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.113112 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd6421ad-3f96-4128-9aae-9bd4ba795a75-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vpdnn\" (UID: \"cd6421ad-3f96-4128-9aae-9bd4ba795a75\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.129168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.129137 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:26.446062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.446039 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn"] Oct 30 07:07:26.446985 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:07:26.446960 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6421ad_3f96_4128_9aae_9bd4ba795a75.slice/crio-b796a5c769ee6b3fdc61113fe0ff2a42a1d8619a0259d9121d89a19efb36283e WatchSource:0}: Error finding container b796a5c769ee6b3fdc61113fe0ff2a42a1d8619a0259d9121d89a19efb36283e: Status 404 returned error can't find the container with id b796a5c769ee6b3fdc61113fe0ff2a42a1d8619a0259d9121d89a19efb36283e Oct 30 07:07:26.449126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.449088 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:26.449233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.449177 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:26.449290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:26.449261 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:07:27.291740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:27.291696 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" event={"ID":"cd6421ad-3f96-4128-9aae-9bd4ba795a75","Type":"ContainerStarted","Data":"63972ffc9a9560509045dbb178bcd506202f3891fb5419afb8a416b05855bdcf"} Oct 30 07:07:27.295787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:27.295737 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" event={"ID":"cd6421ad-3f96-4128-9aae-9bd4ba795a75","Type":"ContainerStarted","Data":"b796a5c769ee6b3fdc61113fe0ff2a42a1d8619a0259d9121d89a19efb36283e"} Oct 30 07:07:27.359021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:27.358968 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podStartSLOduration=2.358953661 podStartE2EDuration="2.358953661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:07:27.358473176 +0000 UTC m=+611.049390613" watchObservedRunningTime="2026-10-30 07:07:27.358953661 +0000 UTC m=+611.049871097" Oct 30 07:07:28.129341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:28.129299 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:28.130931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:28.130904 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:28.131055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:28.130955 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:29.130095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:29.130046 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:29.130596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:29.130110 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:30.129485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:30.129452 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:30.129673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:30.129508 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:31.129810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:31.129773 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:31.130212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:31.129833 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:32.129740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:32.129705 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:32.129923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:32.129760 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:33.129686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:33.129629 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vpdnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 07:07:33.129891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:33.129704 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" podUID="cd6421ad-3f96-4128-9aae-9bd4ba795a75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 07:07:34.720949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:34.720913 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:34.721400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:34.721317 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:34.722059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:34.722040 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vpdnn" Oct 30 07:07:34.833249 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:07:34.833212 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:07:42.750835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:42.750796 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:42.883556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:42.883518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.014026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.013930 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.222693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.222623 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.374659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.374612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.535575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.535537 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.663813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.663726 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:43.957912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:43.957831 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:44.180017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:44.179978 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:44.758890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:44.758847 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:45.302846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:45.302812 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:45.457433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:45.457398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:46.402735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:46.402705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:46.520528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:46.520485 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:46.683306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:46.683229 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:46.842850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:46.842820 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:46.977559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:46.977436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:47.784919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:47.784884 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:47.904826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:47.904791 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:49.108281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:49.108248 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:49.245594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:49.245557 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:49.377729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:49.377596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:49.521684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:49.521648 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:49.762708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:49.762594 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:50.406399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:50.406364 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:50.543857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:50.543820 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:51.141284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:51.141240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:51.260222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:51.260182 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:51.384168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:51.384128 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:51.503028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:51.502936 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:51.622171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:51.622136 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:52.113109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:52.113078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:52.237947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:52.237914 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:52.851653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:52.851600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:52.992490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:52.992454 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:53.112295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:53.112212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:53.642124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:53.642093 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:07:54.745598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:54.745565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:55.531406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:55.531362 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:55.665107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:55.665075 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:56.480333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:56.480295 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:56.599590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:56.599535 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:57.710431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:57.710391 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:07:57.789531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:57.789499 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:57.911021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:57.910984 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:58.232343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:58.232306 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:07:59.709147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:07:59.709111 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:01.231012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:01.230970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:02.959519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:02.959487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:03.080967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:03.080924 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:03.209167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:03.209135 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:04.623387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:04.623349 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:04.829699 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:08:04.829661 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:08:06.225119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.225085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:06.732466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.732432 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv"] Oct 30 07:08:06.736422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.736401 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.767945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.767915 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:08:06.767945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.767923 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:08:06.767945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.767948 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:08:06.768204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.767996 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:08:06.769666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.769646 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv"] Oct 30 07:08:06.897147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.897111 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.897347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.897154 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px24f\" (UniqueName: \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.897347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.897187 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.998166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.998068 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.998166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.998135 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px24f\" (UniqueName: \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.998166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.998157 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:06.998863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:06.998844 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:07.000537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.000514 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:07.023833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.023795 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px24f\" (UniqueName: \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\") pod \"maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:07.052249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.052211 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:07.237500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.237466 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:07.365110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.365080 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv"] Oct 30 07:08:07.366234 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:08:07.366199 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3382115a_ba3e_4f02_aa71_ef2599389f80.slice/crio-d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647 WatchSource:0}: Error finding container d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647: Status 404 returned error can't find the container with id d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647 Oct 30 07:08:07.368304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.368285 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:08:07.496198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.496165 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" event={"ID":"3382115a-ba3e-4f02-aa71-ef2599389f80","Type":"ContainerStarted","Data":"d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647"} Oct 30 07:08:07.901338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:07.901294 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:09.698591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.698552 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:09.755725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.755616 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f"] Oct 30 07:08:09.768264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.768226 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:09.779568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.779318 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f"] Oct 30 07:08:09.929804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.929777 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:09.929943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.929816 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:09.929943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:09.929925 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6pbv\" (UniqueName: \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.031216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.031176 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6pbv\" (UniqueName: \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.031417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.031342 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.031417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.031385 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.032245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.032200 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.034124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.034100 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.057210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.057177 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6pbv\" (UniqueName: \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\") pod \"maas-api-revoke-keys-e2e-ait-37f3d312-ng29f\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.088586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.088556 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:10.398284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.398258 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f"] Oct 30 07:08:10.399547 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:08:10.399510 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72445614_3f01_4689_88b1_e1613f2b68c7.slice/crio-ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b WatchSource:0}: Error finding container ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b: Status 404 returned error can't find the container with id ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b Oct 30 07:08:10.518080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.518037 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" event={"ID":"3382115a-ba3e-4f02-aa71-ef2599389f80","Type":"ContainerStarted","Data":"164aaa60eda99f37a530f5c27c081bc2a47964eb6ee8884782039bcfd7702262"} Oct 30 07:08:10.519279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.519241 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" event={"ID":"72445614-3f01-4689-88b1-e1613f2b68c7","Type":"ContainerStarted","Data":"ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b"} Oct 30 07:08:10.566053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:10.566000 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" podStartSLOduration=2.037345734 podStartE2EDuration="4.565986736s" totalImagesPullingTime="2.528641002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:10.565040004 +0000 UTC m=+654.255957440" watchObservedRunningTime="2026-10-30 07:08:10.565986736 +0000 UTC m=+654.256904173" Oct 30 07:08:11.224746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:11.224709 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:11.525542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:11.525456 5894 generic.go:356] "Generic (PLEG): container finished" podID="3382115a-ba3e-4f02-aa71-ef2599389f80" containerID="164aaa60eda99f37a530f5c27c081bc2a47964eb6ee8884782039bcfd7702262" exitCode=0 Oct 30 07:08:11.525542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:11.525532 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" event={"ID":"3382115a-ba3e-4f02-aa71-ef2599389f80","Type":"ContainerDied","Data":"164aaa60eda99f37a530f5c27c081bc2a47964eb6ee8884782039bcfd7702262"} Oct 30 07:08:11.527096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:11.527069 5894 generic.go:356] "Generic (PLEG): container finished" podID="72445614-3f01-4689-88b1-e1613f2b68c7" containerID="b3768383865da31c3209712b3f1a43fc3d0f88b6d1cb773b77e661c901475edc" exitCode=0 Oct 30 07:08:11.527214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:11.527155 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" event={"ID":"72445614-3f01-4689-88b1-e1613f2b68c7","Type":"ContainerDied","Data":"b3768383865da31c3209712b3f1a43fc3d0f88b6d1cb773b77e661c901475edc"} Oct 30 07:08:12.555889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.555838 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb"] Oct 30 07:08:12.561123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.561093 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.579002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.578915 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb"] Oct 30 07:08:12.659578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.659540 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.659815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.659606 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.659815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.659658 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqdd8\" (UniqueName: \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.717826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.717796 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:12.720880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.720861 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:12.761377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.761344 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.761577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.761386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.761577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.761411 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqdd8\" (UniqueName: \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.762200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.762157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.764541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.764516 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.792079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.791997 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqdd8\" (UniqueName: \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\") pod \"maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.852597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.852563 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx"] Oct 30 07:08:12.853312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853292 5894 cpu_manager.go:414] "containerMap: removing container" podUID="72445614-3f01-4689-88b1-e1613f2b68c7" containerName="revoke-keys" Oct 30 07:08:12.853355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853317 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72445614-3f01-4689-88b1-e1613f2b68c7" containerName="revoke-keys" Oct 30 07:08:12.853391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853370 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3382115a-ba3e-4f02-aa71-ef2599389f80" containerName="revoke-keys" Oct 30 07:08:12.853391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853378 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3382115a-ba3e-4f02-aa71-ef2599389f80" containerName="revoke-keys" Oct 30 07:08:12.853493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853482 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3382115a-ba3e-4f02-aa71-ef2599389f80" containerName="revoke-keys" Oct 30 07:08:12.853527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.853502 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="72445614-3f01-4689-88b1-e1613f2b68c7" containerName="revoke-keys" Oct 30 07:08:12.857272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.857248 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:12.862387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862362 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\") pod \"3382115a-ba3e-4f02-aa71-ef2599389f80\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " Oct 30 07:08:12.862560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862485 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\") pod \"72445614-3f01-4689-88b1-e1613f2b68c7\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " Oct 30 07:08:12.862560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862535 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\") pod \"72445614-3f01-4689-88b1-e1613f2b68c7\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " Oct 30 07:08:12.862719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862592 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\" (UniqueName: \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\") pod \"72445614-3f01-4689-88b1-e1613f2b68c7\" (UID: \"72445614-3f01-4689-88b1-e1613f2b68c7\") " Oct 30 07:08:12.862719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862676 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\") pod \"3382115a-ba3e-4f02-aa71-ef2599389f80\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " Oct 30 07:08:12.862840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862727 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\" (UniqueName: \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\") pod \"3382115a-ba3e-4f02-aa71-ef2599389f80\" (UID: \"3382115a-ba3e-4f02-aa71-ef2599389f80\") " Oct 30 07:08:12.862840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862781 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca" pod "3382115a-ba3e-4f02-aa71-ef2599389f80" (UID: "3382115a-ba3e-4f02-aa71-ef2599389f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:12.862947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.862858 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca" pod "72445614-3f01-4689-88b1-e1613f2b68c7" (UID: "72445614-3f01-4689-88b1-e1613f2b68c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:12.863185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.863152 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.863185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.863180 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.865674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.865595 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca" pod "72445614-3f01-4689-88b1-e1613f2b68c7" (UID: "72445614-3f01-4689-88b1-e1613f2b68c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:12.865915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.865792 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca" pod "3382115a-ba3e-4f02-aa71-ef2599389f80" (UID: "3382115a-ba3e-4f02-aa71-ef2599389f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:12.865915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.865798 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv" pod "72445614-3f01-4689-88b1-e1613f2b68c7" (UID: "72445614-3f01-4689-88b1-e1613f2b68c7"). InnerVolumeSpecName "kube-api-access-q6pbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:12.866051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.865910 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f" pod "3382115a-ba3e-4f02-aa71-ef2599389f80" (UID: "3382115a-ba3e-4f02-aa71-ef2599389f80"). InnerVolumeSpecName "kube-api-access-px24f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:12.874710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.874677 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx"] Oct 30 07:08:12.876268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.876238 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:12.963948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.963800 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8b9\" (UniqueName: \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:12.963948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.963903 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:12.964201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.964140 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:12.964341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.964317 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72445614-3f01-4689-88b1-e1613f2b68c7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.964391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.964349 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6pbv\" (UniqueName: \"kubernetes.io/projected/72445614-3f01-4689-88b1-e1613f2b68c7-kube-api-access-q6pbv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.964391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.964365 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3382115a-ba3e-4f02-aa71-ef2599389f80-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.964391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.964378 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px24f\" (UniqueName: \"kubernetes.io/projected/3382115a-ba3e-4f02-aa71-ef2599389f80-kube-api-access-px24f\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:12.969138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:12.969046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:13.065468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.065387 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.065468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.065448 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8b9\" (UniqueName: \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.065701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.065498 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.066182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.066157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.067977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.067958 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.091252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.091223 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8b9\" (UniqueName: \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\") pod \"maas-api-revoke-keys-e2e-ait-7a77072b-8qngx\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.173544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.173509 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:13.182331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.182307 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb"] Oct 30 07:08:13.185550 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:08:13.185520 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261faf0d_ab0e_41a9_977a_e14f247ab4f8.slice/crio-9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b WatchSource:0}: Error finding container 9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b: Status 404 returned error can't find the container with id 9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b Oct 30 07:08:13.481005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.480979 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx"] Oct 30 07:08:13.481675 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:08:13.481644 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92e4a45_0966_4ca8_8b0f_140356c5147f.slice/crio-288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5 WatchSource:0}: Error finding container 288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5: Status 404 returned error can't find the container with id 288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5 Oct 30 07:08:13.538910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.538873 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" event={"ID":"a92e4a45-0966-4ca8-8b0f-140356c5147f","Type":"ContainerStarted","Data":"288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5"} Oct 30 07:08:13.540379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.540347 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" event={"ID":"261faf0d-ab0e-41a9-977a-e14f247ab4f8","Type":"ContainerStarted","Data":"dd00323a653de78b5a08585f2596d9edef9aba5b183ccbb9a7613936ff83b051"} Oct 30 07:08:13.540522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.540396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" event={"ID":"261faf0d-ab0e-41a9-977a-e14f247ab4f8","Type":"ContainerStarted","Data":"9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b"} Oct 30 07:08:13.541672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.541650 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" Oct 30 07:08:13.541672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.541666 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv" event={"ID":"3382115a-ba3e-4f02-aa71-ef2599389f80","Type":"ContainerDied","Data":"d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647"} Oct 30 07:08:13.541842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.541704 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f3182023195be9b162840d9d78e9bda4076da9daa2fb9a4c48545c4cbe3647" Oct 30 07:08:13.542875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.542853 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" event={"ID":"72445614-3f01-4689-88b1-e1613f2b68c7","Type":"ContainerDied","Data":"ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b"} Oct 30 07:08:13.542984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.542888 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca6e841a1def5cefa4990e24a9833f1ed7896d932410104e934c4ec76203526b" Oct 30 07:08:13.542984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.542861 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f" Oct 30 07:08:13.585412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:13.585358 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" podStartSLOduration=1.585343291 podStartE2EDuration="1.585343291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:08:13.583679509 +0000 UTC m=+657.274596943" watchObservedRunningTime="2026-10-30 07:08:13.585343291 +0000 UTC m=+657.276260728" Oct 30 07:08:14.550518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:14.550433 5894 generic.go:356] "Generic (PLEG): container finished" podID="a92e4a45-0966-4ca8-8b0f-140356c5147f" containerID="5bba9838580a5bd9310e2d597ccf20336b5040ce1baf15807aad84e5a907444e" exitCode=0 Oct 30 07:08:14.550707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:14.550509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" event={"ID":"a92e4a45-0966-4ca8-8b0f-140356c5147f","Type":"ContainerDied","Data":"5bba9838580a5bd9310e2d597ccf20336b5040ce1baf15807aad84e5a907444e"} Oct 30 07:08:14.552119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:14.552092 5894 generic.go:356] "Generic (PLEG): container finished" podID="261faf0d-ab0e-41a9-977a-e14f247ab4f8" containerID="dd00323a653de78b5a08585f2596d9edef9aba5b183ccbb9a7613936ff83b051" exitCode=0 Oct 30 07:08:14.552281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:14.552179 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" event={"ID":"261faf0d-ab0e-41a9-977a-e14f247ab4f8","Type":"ContainerDied","Data":"dd00323a653de78b5a08585f2596d9edef9aba5b183ccbb9a7613936ff83b051"} Oct 30 07:08:14.819574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:14.819481 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:15.504361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.504329 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn"] Oct 30 07:08:15.508464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.508441 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.527761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.527728 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn"] Oct 30 07:08:15.696950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.696916 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.697086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.696967 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smplj\" (UniqueName: \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.697135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.697101 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.705779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.705755 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:15.709130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.709103 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:15.798714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.798350 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.798714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.798401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smplj\" (UniqueName: \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.798714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.798493 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.799169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.799149 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.801374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.801347 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.830467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.830432 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smplj\" (UniqueName: \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\") pod \"maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:15.899399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899369 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\") pod \"a92e4a45-0966-4ca8-8b0f-140356c5147f\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " Oct 30 07:08:15.899605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899436 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\") pod \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " Oct 30 07:08:15.899605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899542 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\") pod \"a92e4a45-0966-4ca8-8b0f-140356c5147f\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " Oct 30 07:08:15.899605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899582 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\" (UniqueName: \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\") pod \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " Oct 30 07:08:15.899825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899665 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\") pod \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\" (UID: \"261faf0d-ab0e-41a9-977a-e14f247ab4f8\") " Oct 30 07:08:15.899825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899719 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\" (UniqueName: \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\") pod \"a92e4a45-0966-4ca8-8b0f-140356c5147f\" (UID: \"a92e4a45-0966-4ca8-8b0f-140356c5147f\") " Oct 30 07:08:15.899929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.899849 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca" pod "a92e4a45-0966-4ca8-8b0f-140356c5147f" (UID: "a92e4a45-0966-4ca8-8b0f-140356c5147f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:15.900122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.900088 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca" pod "261faf0d-ab0e-41a9-977a-e14f247ab4f8" (UID: "261faf0d-ab0e-41a9-977a-e14f247ab4f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:15.900210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.900192 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:15.900245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.900218 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:15.901905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.901880 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9" pod "a92e4a45-0966-4ca8-8b0f-140356c5147f" (UID: "a92e4a45-0966-4ca8-8b0f-140356c5147f"). InnerVolumeSpecName "kube-api-access-2s8b9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:15.901984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.901919 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8" pod "261faf0d-ab0e-41a9-977a-e14f247ab4f8" (UID: "261faf0d-ab0e-41a9-977a-e14f247ab4f8"). InnerVolumeSpecName "kube-api-access-rqdd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:15.902193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.902160 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca" pod "a92e4a45-0966-4ca8-8b0f-140356c5147f" (UID: "a92e4a45-0966-4ca8-8b0f-140356c5147f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:15.902193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:15.902177 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca" pod "261faf0d-ab0e-41a9-977a-e14f247ab4f8" (UID: "261faf0d-ab0e-41a9-977a-e14f247ab4f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:16.001049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.001014 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/261faf0d-ab0e-41a9-977a-e14f247ab4f8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:16.001049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.001045 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92e4a45-0966-4ca8-8b0f-140356c5147f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:16.001049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.001055 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqdd8\" (UniqueName: \"kubernetes.io/projected/261faf0d-ab0e-41a9-977a-e14f247ab4f8-kube-api-access-rqdd8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:16.001300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.001064 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8b9\" (UniqueName: \"kubernetes.io/projected/a92e4a45-0966-4ca8-8b0f-140356c5147f-kube-api-access-2s8b9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:16.121609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.121506 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:16.422547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.422494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:16.425727 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:08:16.425695 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f5bc56e_1c3d_4a4d_b372_2d351bf4d5fb.slice/crio-3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2 WatchSource:0}: Error finding container 3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2: Status 404 returned error can't find the container with id 3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2 Oct 30 07:08:16.432395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.432370 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn"] Oct 30 07:08:16.563376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.563345 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" event={"ID":"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb","Type":"ContainerStarted","Data":"3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2"} Oct 30 07:08:16.564754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.564712 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" event={"ID":"a92e4a45-0966-4ca8-8b0f-140356c5147f","Type":"ContainerDied","Data":"288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5"} Oct 30 07:08:16.564916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.564779 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="288eb0a9f14eed59949a4342221e0d3b10d7deff96c9a8e257a801d8ed19cec5" Oct 30 07:08:16.564916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.564724 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx" Oct 30 07:08:16.566088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.566067 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" event={"ID":"261faf0d-ab0e-41a9-977a-e14f247ab4f8","Type":"ContainerDied","Data":"9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b"} Oct 30 07:08:16.566188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.566104 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb" Oct 30 07:08:16.566188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:16.566107 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eb2f4c459be385bbfcf1082b6f7c90e510cba4a2bb004877ca3d19c5cc9a82b" Oct 30 07:08:17.352394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:17.352357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:17.482409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:17.482372 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:17.574899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:17.574857 5894 generic.go:356] "Generic (PLEG): container finished" podID="8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" containerID="28cbe041f4820839c6e6fa2862ce4cf87c0c4125f3348d8c02eed3efffc2da97" exitCode=0 Oct 30 07:08:17.575095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:17.575032 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" event={"ID":"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb","Type":"ContainerDied","Data":"28cbe041f4820839c6e6fa2862ce4cf87c0c4125f3348d8c02eed3efffc2da97"} Oct 30 07:08:18.148320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.148278 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:18.717283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.717257 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:18.725276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.725252 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\") pod \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " Oct 30 07:08:18.725425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.725293 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\" (UniqueName: \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\") pod \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " Oct 30 07:08:18.725425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.725407 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\") pod \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\" (UID: \"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb\") " Oct 30 07:08:18.725608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.725587 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca" pod "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" (UID: "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:08:18.727451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.727426 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj" pod "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" (UID: "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb"). InnerVolumeSpecName "kube-api-access-smplj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:08:18.727451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.727423 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca" pod "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" (UID: "8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:08:18.826153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.826117 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:18.826153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.826150 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smplj\" (UniqueName: \"kubernetes.io/projected/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-kube-api-access-smplj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:18.826153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:18.826161 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:08:19.422453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.422419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:19.589018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.588992 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" Oct 30 07:08:19.589212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.589030 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn" event={"ID":"8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb","Type":"ContainerDied","Data":"3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2"} Oct 30 07:08:19.589212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.589088 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3680714ca4218d2dd7012b34f22f0bc1d6b6e18d54650686ce149924ca7b8cf2" Oct 30 07:08:19.854932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.854595 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:19.958040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.958008 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv"] Oct 30 07:08:19.960700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.960675 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a5d88019-bgrjv"] Oct 30 07:08:19.971081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:19.971045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:20.092745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:20.090787 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:20.208729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:20.208608 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:20.429795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:20.429763 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3382115a-ba3e-4f02-aa71-ef2599389f80" path="/var/lib/kubelet/pods/3382115a-ba3e-4f02-aa71-ef2599389f80/volumes" Oct 30 07:08:20.655589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:20.655553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:20.768153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:20.768119 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:21.034690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:21.034620 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:21.522599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:21.522497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:21.652390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:21.652332 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:21.834768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:21.834726 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:23.183394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:23.183359 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:23.310216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:23.310181 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:23.424366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:23.424286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:23.529950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:23.529919 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:23.945057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:23.945027 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:24.077579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:24.077541 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:24.806982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:24.806946 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.027799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.027769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.155565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.155464 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.281055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.281028 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.440066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.439983 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx"] Oct 30 07:08:25.453350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.453318 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7a77072b-8qngx"] Oct 30 07:08:25.463996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.463958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.637774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.637744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.767194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.767116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:25.846657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.846600 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb"] Oct 30 07:08:25.860601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.860570 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9bd1c95c-cl2nb"] Oct 30 07:08:25.890836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:25.890800 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.049393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.049356 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.161969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.161935 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.250160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.250129 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f"] Oct 30 07:08:26.254057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.254033 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-37f3d312-ng29f"] Oct 30 07:08:26.286485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.286444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.401864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.401777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.431608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.431572 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261faf0d-ab0e-41a9-977a-e14f247ab4f8" path="/var/lib/kubelet/pods/261faf0d-ab0e-41a9-977a-e14f247ab4f8/volumes" Oct 30 07:08:26.432105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.432088 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72445614-3f01-4689-88b1-e1613f2b68c7" path="/var/lib/kubelet/pods/72445614-3f01-4689-88b1-e1613f2b68c7/volumes" Oct 30 07:08:26.432448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.432436 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92e4a45-0966-4ca8-8b0f-140356c5147f" path="/var/lib/kubelet/pods/a92e4a45-0966-4ca8-8b0f-140356c5147f/volumes" Oct 30 07:08:26.522883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.522852 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.641005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.640972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:26.922418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:26.922339 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.047306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.047274 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.162338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.162302 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.398515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.398469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.557128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.557097 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.677104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.677027 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.838260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.838228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:27.952433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:27.952353 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:28.071503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:28.071469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:28.321294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:28.321262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:28.452332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:28.452298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:28.600859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:28.600709 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:28.809723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:28.809688 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:29.716162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:29.716133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:29.828261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:29.828221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:31.167465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.167429 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:31.441286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.441201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:31.661182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.661142 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:31.844292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.844250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:31.907198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.907164 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn"] Oct 30 07:08:31.920528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.920501 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-77bbf25a-rrpwn"] Oct 30 07:08:31.967511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:31.967473 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:32.118336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.118249 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:32.264625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.264591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:32.421375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.421293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:32.430264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.430237 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" path="/var/lib/kubelet/pods/8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb/volumes" Oct 30 07:08:32.726205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.726111 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:32.982195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:32.982100 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:33.171819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:33.171789 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:33.435157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:33.435126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:33.623559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:33.623529 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:33.747664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:33.747548 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:34.021264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:34.021177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:34.830597 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:08:34.830560 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:08:35.098922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:35.098815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:35.230416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:35.230383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:35.677527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:35.677477 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:35.795005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:35.794976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:35.913885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:35.913842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:36.098108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:36.098075 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:36.215085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:36.215052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:37.014782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:37.014714 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:37.154821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:37.154777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:37.294086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:37.294038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:37.527080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:37.527042 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:37.642842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:37.642751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:39.293618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:39.293585 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:39.412044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:39.411998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:40.988296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:40.988258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:41.805775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:41.805744 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:42.580804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:42.580767 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:44.315585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:44.315546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:44.430785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:44.430748 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:45.090665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:45.090613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:45.233729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:45.233695 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:45.929443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:45.929408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:46.062453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:46.062416 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:46.328840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:46.328805 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:47.919498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:47.919454 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:48.140139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:48.140106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:49.240453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:49.240417 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:49.632565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:49.632533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:51.309389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:51.309357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:51.424999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:51.424965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:51.890617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:51.890582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.071417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.071383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.194413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.194324 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.341474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.341441 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.556179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.556145 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.681913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.681874 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.800024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.799955 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:52.993072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:52.992971 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:53.296354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:53.296317 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:54.810608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:54.810559 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:55.635865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:55.635825 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:08:56.358627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:56.358591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:57.039768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:57.039734 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:57.158703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:57.158627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:57.711621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:57.711589 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:57.871257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:57.871219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:58.006209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:58.006126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:58.124558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:58.124526 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:08:59.715514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:08:59.715461 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:01.309597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:01.309563 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:01.675269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:01.675174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:02.911556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:02.911518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:03.697794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:03.697731 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:04.421575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:04.421539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:04.831021 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:09:04.830984 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:09:05.908806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:05.908762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:06.068391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:06.068350 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:07.321562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:07.321524 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:09:07.500787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:07.500754 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:09.197694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:09.197624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:09.383919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:09.383883 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:10.783551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:10.783515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:12.301199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:12.301166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:13.773692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:13.773656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:15.502932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:15.502893 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:16.893536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:16.893495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:18.290346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:18.290313 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:09:18.447401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:18.447367 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:19.973246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:19.973210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:21.435890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:21.435851 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:22.107421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:22.107383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:22.993003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:22.992968 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:24.446505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:24.446470 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:25.427977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:25.427945 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:26.132149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:26.132112 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:27.713226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:27.713131 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:29.225577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:29.225541 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:30.718907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.718865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:30.925758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.925721 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 07:09:30.926746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926717 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a92e4a45-0966-4ca8-8b0f-140356c5147f" containerName="revoke-keys" Oct 30 07:09:30.926746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926743 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92e4a45-0966-4ca8-8b0f-140356c5147f" containerName="revoke-keys" Oct 30 07:09:30.926944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926764 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" containerName="revoke-keys" Oct 30 07:09:30.926944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926772 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" containerName="revoke-keys" Oct 30 07:09:30.926944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926814 5894 cpu_manager.go:414] "containerMap: removing container" podUID="261faf0d-ab0e-41a9-977a-e14f247ab4f8" containerName="revoke-keys" Oct 30 07:09:30.926944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926822 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261faf0d-ab0e-41a9-977a-e14f247ab4f8" containerName="revoke-keys" Oct 30 07:09:30.927074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926961 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92e4a45-0966-4ca8-8b0f-140356c5147f" containerName="revoke-keys" Oct 30 07:09:30.927074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926972 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f5bc56e-1c3d-4a4d-b372-2d351bf4d5fb" containerName="revoke-keys" Oct 30 07:09:30.927074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.926987 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="261faf0d-ab0e-41a9-977a-e14f247ab4f8" containerName="revoke-keys" Oct 30 07:09:30.930256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.930235 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.949708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.949682 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 07:09:30.961905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.961878 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"default-dockercfg-7kb4d\"" Oct 30 07:09:30.962081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.961978 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"premium-w3-2f845b-kserve-self-signed-certs\"" Oct 30 07:09:30.964347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.964324 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"openshift-service-ca.crt\"" Oct 30 07:09:30.964464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.964408 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"kube-root-ca.crt\"" Oct 30 07:09:30.995754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.995671 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.995973 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w425\" (UniqueName: \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.996019 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.996112 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.996245 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.996318 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:30.996538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:30.996509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:31.097833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.097792 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.097833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.097835 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w425\" (UniqueName: \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.097866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.097902 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.097998 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.098061 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.098334 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.098811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.098787 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.099079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.099058 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.101213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.101185 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.102185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.102150 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.124404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.124339 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w425\" (UniqueName: \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\") pod \"premium-w3-2f845b-kserve-676d96cc4-tv8t5\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.243279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.243244 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:31.549295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:31.549270 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 07:09:31.550087 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:09:31.550049 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod973f8c38_94b3_4869_8df0_2691c6ecb509.slice/crio-10516d74d1471281d5052d25d661c23de0fc2accd1d4d117ecf8d21a8b39a0ec WatchSource:0}: Error finding container 10516d74d1471281d5052d25d661c23de0fc2accd1d4d117ecf8d21a8b39a0ec: Status 404 returned error can't find the container with id 10516d74d1471281d5052d25d661c23de0fc2accd1d4d117ecf8d21a8b39a0ec Oct 30 07:09:32.047457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:32.047414 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" event={"ID":"973f8c38-94b3-4869-8df0-2691c6ecb509","Type":"ContainerStarted","Data":"10516d74d1471281d5052d25d661c23de0fc2accd1d4d117ecf8d21a8b39a0ec"} Oct 30 07:09:32.182884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:32.182849 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:09:32.245181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:32.245147 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:33.054272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:33.054229 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" event={"ID":"973f8c38-94b3-4869-8df0-2691c6ecb509","Type":"ContainerStarted","Data":"ad9b7c6904a999bacaf63df19e785f2502e64852b7d62e08383606dc5a59ea1c"} Oct 30 07:09:33.104727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:33.104671 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podStartSLOduration=1.997666232 podStartE2EDuration="3.104650696s" totalImagesPullingTime="1.106984464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:33.102064928 +0000 UTC m=+736.792982370" watchObservedRunningTime="2026-10-30 07:09:33.104650696 +0000 UTC m=+736.795568127" Oct 30 07:09:33.244399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:33.244363 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:33.908052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:33.908017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:34.831156 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:09:34.831116 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:09:35.419145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:35.419071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:35.547499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:35.547460 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:37.034847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:37.034811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:38.620179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:38.620142 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:39.044061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.044021 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 07:09:39.047931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.047909 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.080845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.080817 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"openshift-service-ca.crt\"" Oct 30 07:09:39.081043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.080880 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"premium-w2-e5fa2d-kserve-self-signed-certs\"" Oct 30 07:09:39.081043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.080839 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"default-dockercfg-r2klx\"" Oct 30 07:09:39.081043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.080818 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"kube-root-ca.crt\"" Oct 30 07:09:39.082468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.082444 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 07:09:39.102935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.102897 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:39.187608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.187575 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.187608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.187623 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.187872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.187714 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvpc\" (UniqueName: \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.187872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.187777 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.188008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.187982 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.188047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.188032 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289077 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289126 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289166 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289199 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289226 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvpc\" (UniqueName: \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289251 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289586 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289628 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.289789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.289741 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.291879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.291847 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.292034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.292014 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.314902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.314831 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvpc\" (UniqueName: \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\") pod \"premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.361522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.361481 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:39.668322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:39.668290 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 07:09:39.669672 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:09:39.669626 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fa6b8eb_4082_4703_aa04_7a3d8165a505.slice/crio-762bc8bb4f459e0b8bad6dcb84e358739147338f81dcd69679c96291d74e3216 WatchSource:0}: Error finding container 762bc8bb4f459e0b8bad6dcb84e358739147338f81dcd69679c96291d74e3216: Status 404 returned error can't find the container with id 762bc8bb4f459e0b8bad6dcb84e358739147338f81dcd69679c96291d74e3216 Oct 30 07:09:40.092862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:40.092821 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" event={"ID":"9fa6b8eb-4082-4703-aa04-7a3d8165a505","Type":"ContainerStarted","Data":"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530"} Oct 30 07:09:40.093037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:40.092871 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" event={"ID":"9fa6b8eb-4082-4703-aa04-7a3d8165a505","Type":"ContainerStarted","Data":"762bc8bb4f459e0b8bad6dcb84e358739147338f81dcd69679c96291d74e3216"} Oct 30 07:09:40.138575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:40.138515 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" podStartSLOduration=1.138500218 podStartE2EDuration="1.138500218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:09:40.137596996 +0000 UTC m=+743.828514433" watchObservedRunningTime="2026-10-30 07:09:40.138500218 +0000 UTC m=+743.829417652" Oct 30 07:09:40.224474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:40.224438 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:40.362336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:40.362255 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:41.244311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:41.244276 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:41.257301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:41.257269 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:41.818790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:41.818753 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:42.115536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:42.115450 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:09:42.399109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:42.399031 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:09:43.522577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:43.522533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:45.115242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:45.115210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:45.241602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:45.241569 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:46.633510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:46.633467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:46.747764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:46.747731 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:48.324798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:48.324765 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:49.362606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:49.362575 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:49.375672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:49.375643 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:49.813662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:49.813618 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:50.156844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:50.156768 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 07:09:51.494243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:51.494208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:53.101585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:53.101538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:54.516316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:54.516275 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:54.691660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:54.691600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:54.806165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:54.806128 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:56.218404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:56.218362 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:56.938664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:56.938594 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:09:57.896048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:57.896012 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:09:59.521509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:09:59.521473 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:01.012765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:01.012728 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:02.653121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:02.653074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:04.205602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:04.205551 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:04.828486 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:10:04.828446 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:10:05.706723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:05.706681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:07.218194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:07.218154 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:08.837740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:08.837707 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:09.727445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:09.727409 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:10.628157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:10.628121 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:11.281059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.281031 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc"] Oct 30 07:10:11.284813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.284790 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.302310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.302280 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:10:11.311855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.311827 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.312002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.311967 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.312044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.312013 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b56mf\" (UniqueName: \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.316317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.316291 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:10:11.316317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.316291 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:10:11.316499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.316473 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:10:11.318087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.318061 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc"] Oct 30 07:10:11.412744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.412701 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.412952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.412804 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.412952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.412826 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b56mf\" (UniqueName: \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.413569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.413529 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.415645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.415608 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.438132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.438095 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b56mf\" (UniqueName: \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\") pod \"maas-api-revoke-sub-f268271d-tt9tc\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.597992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.597909 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:11.903235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:11.903206 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc"] Oct 30 07:10:11.904728 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:11.904698 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba58660_f309_45f9_9226_4921ecd9153d.slice/crio-178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2 WatchSource:0}: Error finding container 178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2: Status 404 returned error can't find the container with id 178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2 Oct 30 07:10:12.268215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:12.268169 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" event={"ID":"2ba58660-f309-45f9-9226-4921ecd9153d","Type":"ContainerStarted","Data":"5fe7dfa342e35a453c97d2b4ef017c9d4edcfd3e59e0007a989fe8332fb0ce98"} Oct 30 07:10:12.268414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:12.268274 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" event={"ID":"2ba58660-f309-45f9-9226-4921ecd9153d","Type":"ContainerStarted","Data":"178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2"} Oct 30 07:10:12.292610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:12.292575 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:12.315443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:12.315387 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" podStartSLOduration=1.315374308 podStartE2EDuration="1.315374308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:12.314889963 +0000 UTC m=+776.005807402" watchObservedRunningTime="2026-10-30 07:10:12.315374308 +0000 UTC m=+776.006291745" Oct 30 07:10:13.275357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:13.275269 5894 generic.go:356] "Generic (PLEG): container finished" podID="2ba58660-f309-45f9-9226-4921ecd9153d" containerID="5fe7dfa342e35a453c97d2b4ef017c9d4edcfd3e59e0007a989fe8332fb0ce98" exitCode=0 Oct 30 07:10:13.275357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:13.275337 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" event={"ID":"2ba58660-f309-45f9-9226-4921ecd9153d","Type":"ContainerDied","Data":"5fe7dfa342e35a453c97d2b4ef017c9d4edcfd3e59e0007a989fe8332fb0ce98"} Oct 30 07:10:13.542594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:13.542562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:13.877188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:13.877102 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:14.424760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.424734 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:14.546853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.546774 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\") pod \"2ba58660-f309-45f9-9226-4921ecd9153d\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " Oct 30 07:10:14.546853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.546825 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\" (UniqueName: \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\") pod \"2ba58660-f309-45f9-9226-4921ecd9153d\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " Oct 30 07:10:14.547101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.546904 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\") pod \"2ba58660-f309-45f9-9226-4921ecd9153d\" (UID: \"2ba58660-f309-45f9-9226-4921ecd9153d\") " Oct 30 07:10:14.547216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.547183 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca" pod "2ba58660-f309-45f9-9226-4921ecd9153d" (UID: "2ba58660-f309-45f9-9226-4921ecd9153d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:14.549216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.549185 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca" pod "2ba58660-f309-45f9-9226-4921ecd9153d" (UID: "2ba58660-f309-45f9-9226-4921ecd9153d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:14.549345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.549193 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf" pod "2ba58660-f309-45f9-9226-4921ecd9153d" (UID: "2ba58660-f309-45f9-9226-4921ecd9153d"). InnerVolumeSpecName "kube-api-access-b56mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:14.648382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.648344 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:14.648382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.648372 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b56mf\" (UniqueName: \"kubernetes.io/projected/2ba58660-f309-45f9-9226-4921ecd9153d-kube-api-access-b56mf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:14.648382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:14.648382 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ba58660-f309-45f9-9226-4921ecd9153d-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:15.287422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:15.287389 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" Oct 30 07:10:15.287601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:15.287414 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc" event={"ID":"2ba58660-f309-45f9-9226-4921ecd9153d","Type":"ContainerDied","Data":"178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2"} Oct 30 07:10:15.287601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:15.287473 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178204942dcca0b92c54fa4f6fc324935793bb82adf34a351a22901f23985ce2" Oct 30 07:10:15.330668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:15.330609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:17.007087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:17.007043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:18.505384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:18.505337 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:20.103607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:20.103552 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:21.713505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:21.713465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:22.899720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:22.899684 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:10:23.221557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:23.221472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:24.694701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:24.694625 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:25.557051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:25.557018 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:26.211409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.211378 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:26.383556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.383521 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv"] Oct 30 07:10:26.384237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.384218 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2ba58660-f309-45f9-9226-4921ecd9153d" containerName="revoke-keys" Oct 30 07:10:26.384291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.384240 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba58660-f309-45f9-9226-4921ecd9153d" containerName="revoke-keys" Oct 30 07:10:26.384386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.384373 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba58660-f309-45f9-9226-4921ecd9153d" containerName="revoke-keys" Oct 30 07:10:26.388017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.387998 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.406116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.406080 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:10:26.419864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.419833 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:10:26.419864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.419833 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:10:26.420066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.419882 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:10:26.421416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.421395 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv"] Oct 30 07:10:26.468378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.468281 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.468378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.468327 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.468668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.468518 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfg22\" (UniqueName: \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.569138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.569099 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.569336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.569145 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.569336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.569273 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfg22\" (UniqueName: \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.569865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.569840 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.571434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.571414 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.595292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.595261 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfg22\" (UniqueName: \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\") pod \"maas-api-revoke-sub-862e9d30-rpdjv\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:26.701469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:26.701431 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:27.003547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:27.003514 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv"] Oct 30 07:10:27.005189 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:27.005134 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610e6c9e_be60_4e46_a64f_d281a6ee59b1.slice/crio-e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf WatchSource:0}: Error finding container e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf: Status 404 returned error can't find the container with id e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf Oct 30 07:10:27.351585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:27.351545 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" event={"ID":"610e6c9e-be60-4e46-a64f-d281a6ee59b1","Type":"ContainerStarted","Data":"e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf"} Oct 30 07:10:27.697304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:27.697216 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:28.358281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:28.358250 5894 generic.go:356] "Generic (PLEG): container finished" podID="610e6c9e-be60-4e46-a64f-d281a6ee59b1" containerID="f3776ba761b79397f4aabb9cde3503c7f8498645ee7de2df1b84213f1f777c2d" exitCode=0 Oct 30 07:10:28.358719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:28.358337 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" event={"ID":"610e6c9e-be60-4e46-a64f-d281a6ee59b1","Type":"ContainerDied","Data":"f3776ba761b79397f4aabb9cde3503c7f8498645ee7de2df1b84213f1f777c2d"} Oct 30 07:10:29.239849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.239815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:29.509681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.509620 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:29.603488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.603456 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\" (UniqueName: \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\") pod \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " Oct 30 07:10:29.603711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.603517 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\") pod \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " Oct 30 07:10:29.603711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.603547 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\") pod \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\" (UID: \"610e6c9e-be60-4e46-a64f-d281a6ee59b1\") " Oct 30 07:10:29.604045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.603933 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca" pod "610e6c9e-be60-4e46-a64f-d281a6ee59b1" (UID: "610e6c9e-be60-4e46-a64f-d281a6ee59b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:29.604262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.604241 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:29.605825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.605793 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca" pod "610e6c9e-be60-4e46-a64f-d281a6ee59b1" (UID: "610e6c9e-be60-4e46-a64f-d281a6ee59b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:29.605908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.605855 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22" pod "610e6c9e-be60-4e46-a64f-d281a6ee59b1" (UID: "610e6c9e-be60-4e46-a64f-d281a6ee59b1"). InnerVolumeSpecName "kube-api-access-lfg22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:29.704880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.704846 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfg22\" (UniqueName: \"kubernetes.io/projected/610e6c9e-be60-4e46-a64f-d281a6ee59b1-kube-api-access-lfg22\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:29.704880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:29.704876 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/610e6c9e-be60-4e46-a64f-d281a6ee59b1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:30.373096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:30.373063 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" Oct 30 07:10:30.373275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:30.373093 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv" event={"ID":"610e6c9e-be60-4e46-a64f-d281a6ee59b1","Type":"ContainerDied","Data":"e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf"} Oct 30 07:10:30.373275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:30.373141 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e97e07e804a24bb6d11bf71d29e047071b3127ea42a636d18b05dc5b8f3ecadf" Oct 30 07:10:30.714144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:30.714063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:32.328366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:32.328323 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:33.921585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:33.921546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:34.829149 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:10:34.829109 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:10:35.428160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:35.428122 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:37.004564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:37.004501 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:38.508906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:38.508872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:39.275181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.275145 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm"] Oct 30 07:10:39.275835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.275816 5894 cpu_manager.go:414] "containerMap: removing container" podUID="610e6c9e-be60-4e46-a64f-d281a6ee59b1" containerName="revoke-keys" Oct 30 07:10:39.275900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.275837 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="610e6c9e-be60-4e46-a64f-d281a6ee59b1" containerName="revoke-keys" Oct 30 07:10:39.275984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.275973 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="610e6c9e-be60-4e46-a64f-d281a6ee59b1" containerName="revoke-keys" Oct 30 07:10:39.279738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.279714 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.298360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.298327 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm"] Oct 30 07:10:39.299254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.299235 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.299322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.299294 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.299358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.299350 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcx26\" (UniqueName: \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.311038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.310999 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:10:39.311038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.311024 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:10:39.311038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.310999 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:10:39.311270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.311092 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:10:39.399858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.399826 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcx26\" (UniqueName: \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.400020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.399913 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.400020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.399972 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.400730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.400709 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.402161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.402138 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.428737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.428707 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcx26\" (UniqueName: \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\") pod \"maas-api-revoke-sub-f4ab7a28-mcrkm\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.592967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.592929 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:39.893572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:39.893537 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm"] Oct 30 07:10:39.894555 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:39.894518 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5fac62_6f0d_4b2d_bb81_f1ed38559cd3.slice/crio-2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0 WatchSource:0}: Error finding container 2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0: Status 404 returned error can't find the container with id 2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0 Oct 30 07:10:40.027689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:40.027629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:40.434423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:40.434342 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" event={"ID":"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3","Type":"ContainerStarted","Data":"0078a117151a0b3582dcf42270b47fe6332c5413d5fc370909240e6026b9ee6c"} Oct 30 07:10:40.434423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:40.434393 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" event={"ID":"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3","Type":"ContainerStarted","Data":"2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0"} Oct 30 07:10:40.479664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:40.479597 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" podStartSLOduration=1.479582205 podStartE2EDuration="1.479582205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:40.47838022 +0000 UTC m=+804.169297657" watchObservedRunningTime="2026-10-30 07:10:40.479582205 +0000 UTC m=+804.170499641" Oct 30 07:10:41.331650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.331602 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9"] Oct 30 07:10:41.335506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.335484 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.353504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.353469 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9"] Oct 30 07:10:41.396984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.396954 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h"] Oct 30 07:10:41.400743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.400726 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.419814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.419788 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h"] Oct 30 07:10:41.421041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421021 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.421139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421060 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62v6\" (UniqueName: \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.421139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421080 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.421241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421229 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm8m\" (UniqueName: \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.421276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421262 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.421415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.421392 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.442714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.442675 5894 generic.go:356] "Generic (PLEG): container finished" podID="3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" containerID="0078a117151a0b3582dcf42270b47fe6332c5413d5fc370909240e6026b9ee6c" exitCode=0 Oct 30 07:10:41.442899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.442722 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" event={"ID":"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3","Type":"ContainerDied","Data":"0078a117151a0b3582dcf42270b47fe6332c5413d5fc370909240e6026b9ee6c"} Oct 30 07:10:41.510859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.510827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:41.522188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522155 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.522377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522238 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62v6\" (UniqueName: \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.522377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522268 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.522502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522414 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm8m\" (UniqueName: \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.522502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522475 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.522621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.522577 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.523030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.523008 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.523364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.523341 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.524811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.524793 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.524939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.524917 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.549908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.549008 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62v6\" (UniqueName: \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\") pod \"maas-api-revoke-sub-a5f9fc42-mwv9h\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.553429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.553397 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm8m\" (UniqueName: \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\") pod \"maas-api-revoke-sub-d187f494-s5jp9\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.650030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.649947 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:41.715138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.715081 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:41.733282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.733251 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n"] Oct 30 07:10:41.737479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.737455 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.759262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.758969 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n"] Oct 30 07:10:41.825428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.825386 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.825428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.825434 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.825720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.825516 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccqp8\" (UniqueName: \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.926191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.926106 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.926191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.926153 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.926191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.926177 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccqp8\" (UniqueName: \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.926906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.926882 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.928395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.928375 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.952379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.952344 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccqp8\" (UniqueName: \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\") pod \"maas-api-revoke-sub-ba63d382-zkv2n\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:41.952957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:41.952933 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9"] Oct 30 07:10:41.953211 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:41.953183 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a5f763_fdce_430e_bb11_32f3e34fdc97.slice/crio-b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd WatchSource:0}: Error finding container b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd: Status 404 returned error can't find the container with id b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd Oct 30 07:10:42.030661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.030619 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h"] Oct 30 07:10:42.031459 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:42.031434 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dad459d_9254_42a0_8094_3c1f29a98553.slice/crio-ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853 WatchSource:0}: Error finding container ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853: Status 404 returned error can't find the container with id ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853 Oct 30 07:10:42.055331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.055303 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:42.358431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.358406 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n"] Oct 30 07:10:42.359467 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:42.359443 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8d6a99_6390_4f97_b4fd_44a2cf84cce7.slice/crio-5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa WatchSource:0}: Error finding container 5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa: Status 404 returned error can't find the container with id 5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa Oct 30 07:10:42.450305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.450212 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" event={"ID":"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7","Type":"ContainerStarted","Data":"5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa"} Oct 30 07:10:42.452122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.452090 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" event={"ID":"04a5f763-fdce-430e-bb11-32f3e34fdc97","Type":"ContainerStarted","Data":"c2d3e2f91148883d66d19dcc3861597d74c856e82a62f8410886dc06e485eecf"} Oct 30 07:10:42.452287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.452142 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" event={"ID":"04a5f763-fdce-430e-bb11-32f3e34fdc97","Type":"ContainerStarted","Data":"b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd"} Oct 30 07:10:42.453658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.453610 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" event={"ID":"3dad459d-9254-42a0-8094-3c1f29a98553","Type":"ContainerStarted","Data":"6f4d1bae568feec5770a2adc8f1a108e99c4b332cea3d1e78f18ff2936e68044"} Oct 30 07:10:42.453765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.453678 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" event={"ID":"3dad459d-9254-42a0-8094-3c1f29a98553","Type":"ContainerStarted","Data":"ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853"} Oct 30 07:10:42.500420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.500369 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" podStartSLOduration=1.500353681 podStartE2EDuration="1.500353681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:42.499625385 +0000 UTC m=+806.190542823" watchObservedRunningTime="2026-10-30 07:10:42.500353681 +0000 UTC m=+806.191271117" Oct 30 07:10:42.544854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.544791 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" podStartSLOduration=1.544768068 podStartE2EDuration="1.544768068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:42.544325046 +0000 UTC m=+806.235242483" watchObservedRunningTime="2026-10-30 07:10:42.544768068 +0000 UTC m=+806.235685507" Oct 30 07:10:42.616539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.616511 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:42.638346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.635278 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\") pod \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " Oct 30 07:10:42.638346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.635892 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\") pod \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " Oct 30 07:10:42.638346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.635914 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca" pod "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" (UID: "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:42.646103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.638861 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\" (UniqueName: \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\") pod \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\" (UID: \"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3\") " Oct 30 07:10:42.646103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.639368 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:42.673057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.673015 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca" pod "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" (UID: "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:42.673224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.673102 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26" pod "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" (UID: "3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3"). InnerVolumeSpecName "kube-api-access-xcx26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:42.744299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.740456 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:42.744299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:42.740496 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcx26\" (UniqueName: \"kubernetes.io/projected/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3-kube-api-access-xcx26\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:43.023780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.023690 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:43.460172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.460135 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" containerID="8dac2972b9d3383a273dc5ec135cf5cfcadb2f823cfda97b092ac1da5913e619" exitCode=0 Oct 30 07:10:43.460663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.460226 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" event={"ID":"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7","Type":"ContainerDied","Data":"8dac2972b9d3383a273dc5ec135cf5cfcadb2f823cfda97b092ac1da5913e619"} Oct 30 07:10:43.461884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.461858 5894 generic.go:356] "Generic (PLEG): container finished" podID="04a5f763-fdce-430e-bb11-32f3e34fdc97" containerID="c2d3e2f91148883d66d19dcc3861597d74c856e82a62f8410886dc06e485eecf" exitCode=0 Oct 30 07:10:43.461998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.461937 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" event={"ID":"04a5f763-fdce-430e-bb11-32f3e34fdc97","Type":"ContainerDied","Data":"c2d3e2f91148883d66d19dcc3861597d74c856e82a62f8410886dc06e485eecf"} Oct 30 07:10:43.463208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.463190 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" Oct 30 07:10:43.463208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.463199 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm" event={"ID":"3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3","Type":"ContainerDied","Data":"2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0"} Oct 30 07:10:43.463358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.463236 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c051fdd1f837d55792a526e0720c1fc6dca66dc8c75abf995d74b4fb5fb6bf0" Oct 30 07:10:43.464755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.464733 5894 generic.go:356] "Generic (PLEG): container finished" podID="3dad459d-9254-42a0-8094-3c1f29a98553" containerID="6f4d1bae568feec5770a2adc8f1a108e99c4b332cea3d1e78f18ff2936e68044" exitCode=0 Oct 30 07:10:43.464840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:43.464798 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" event={"ID":"3dad459d-9254-42a0-8094-3c1f29a98553","Type":"ContainerDied","Data":"6f4d1bae568feec5770a2adc8f1a108e99c4b332cea3d1e78f18ff2936e68044"} Oct 30 07:10:44.630671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.629359 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:44.659995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.659972 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\" (UniqueName: \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\") pod \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " Oct 30 07:10:44.660119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.660086 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\") pod \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " Oct 30 07:10:44.660168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.660156 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\") pod \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\" (UID: \"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7\") " Oct 30 07:10:44.660501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.660475 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca" pod "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" (UID: "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:44.662349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.662308 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8" pod "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" (UID: "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7"). InnerVolumeSpecName "kube-api-access-ccqp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:44.662349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.662330 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca" pod "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" (UID: "4b8d6a99-6390-4f97-b4fd-44a2cf84cce7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:44.665070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.665033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:44.683268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.683238 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:44.685248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.685227 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:44.761543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761452 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\") pod \"04a5f763-fdce-430e-bb11-32f3e34fdc97\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " Oct 30 07:10:44.761543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761520 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\") pod \"3dad459d-9254-42a0-8094-3c1f29a98553\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " Oct 30 07:10:44.761813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761582 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\" (UniqueName: \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\") pod \"3dad459d-9254-42a0-8094-3c1f29a98553\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " Oct 30 07:10:44.761813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761611 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\") pod \"3dad459d-9254-42a0-8094-3c1f29a98553\" (UID: \"3dad459d-9254-42a0-8094-3c1f29a98553\") " Oct 30 07:10:44.761813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761707 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\" (UniqueName: \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\") pod \"04a5f763-fdce-430e-bb11-32f3e34fdc97\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " Oct 30 07:10:44.761813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.761758 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\") pod \"04a5f763-fdce-430e-bb11-32f3e34fdc97\" (UID: \"04a5f763-fdce-430e-bb11-32f3e34fdc97\") " Oct 30 07:10:44.762423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.762133 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.762423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.762157 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.762423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.762169 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccqp8\" (UniqueName: \"kubernetes.io/projected/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7-kube-api-access-ccqp8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.762423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.762232 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca" pod "3dad459d-9254-42a0-8094-3c1f29a98553" (UID: "3dad459d-9254-42a0-8094-3c1f29a98553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:44.762656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.762489 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca" pod "04a5f763-fdce-430e-bb11-32f3e34fdc97" (UID: "04a5f763-fdce-430e-bb11-32f3e34fdc97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:10:44.764214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.764171 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6" pod "3dad459d-9254-42a0-8094-3c1f29a98553" (UID: "3dad459d-9254-42a0-8094-3c1f29a98553"). InnerVolumeSpecName "kube-api-access-n62v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:44.764308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.764257 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca" pod "3dad459d-9254-42a0-8094-3c1f29a98553" (UID: "3dad459d-9254-42a0-8094-3c1f29a98553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:44.764429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.764406 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca" pod "04a5f763-fdce-430e-bb11-32f3e34fdc97" (UID: "04a5f763-fdce-430e-bb11-32f3e34fdc97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:10:44.764537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.764516 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m" pod "04a5f763-fdce-430e-bb11-32f3e34fdc97" (UID: "04a5f763-fdce-430e-bb11-32f3e34fdc97"). InnerVolumeSpecName "kube-api-access-6gm8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:10:44.863431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863398 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.863431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863429 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04a5f763-fdce-430e-bb11-32f3e34fdc97-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.863678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863446 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.863678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863457 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62v6\" (UniqueName: \"kubernetes.io/projected/3dad459d-9254-42a0-8094-3c1f29a98553-kube-api-access-n62v6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.863678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863469 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad459d-9254-42a0-8094-3c1f29a98553-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:44.863678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:44.863481 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm8m\" (UniqueName: \"kubernetes.io/projected/04a5f763-fdce-430e-bb11-32f3e34fdc97-kube-api-access-6gm8m\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:10:45.476135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.476103 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" Oct 30 07:10:45.476326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.476135 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n" event={"ID":"4b8d6a99-6390-4f97-b4fd-44a2cf84cce7","Type":"ContainerDied","Data":"5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa"} Oct 30 07:10:45.476326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.476193 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a9724145633fc0d158c35544c46d60b6d39a0fa1d51e0dc7e7af0318d1512fa" Oct 30 07:10:45.477501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.477473 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" Oct 30 07:10:45.477626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.477520 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9" event={"ID":"04a5f763-fdce-430e-bb11-32f3e34fdc97","Type":"ContainerDied","Data":"b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd"} Oct 30 07:10:45.477626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.477561 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5480caec505798b9886e5e9f15ccea57ad0b9d98dbb6832e3a9ce9ec9e75abd" Oct 30 07:10:45.478947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.478931 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" Oct 30 07:10:45.479035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.478933 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h" event={"ID":"3dad459d-9254-42a0-8094-3c1f29a98553","Type":"ContainerDied","Data":"ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853"} Oct 30 07:10:45.479035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:45.479027 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccf92e5fc21dff383257183dd1b96f6427375b4218a2e90f452d597b37fe8853" Oct 30 07:10:46.134464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:46.134430 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:47.705223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:47.705192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:49.291625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:49.291588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:50.900575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:50.900541 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:52.499069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:52.499025 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:54.186249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:54.186215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:55.821018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.820985 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:55.876600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.876566 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:10:55.877171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877152 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" containerName="revoke-keys" Oct 30 07:10:55.877171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877170 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877184 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877189 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877203 5894 cpu_manager.go:414] "containerMap: removing container" podUID="04a5f763-fdce-430e-bb11-32f3e34fdc97" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877208 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a5f763-fdce-430e-bb11-32f3e34fdc97" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877225 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3dad459d-9254-42a0-8094-3c1f29a98553" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877231 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dad459d-9254-42a0-8094-3c1f29a98553" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877323 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877336 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877348 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a5f763-fdce-430e-bb11-32f3e34fdc97" containerName="revoke-keys" Oct 30 07:10:55.877354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.877361 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dad459d-9254-42a0-8094-3c1f29a98553" containerName="revoke-keys" Oct 30 07:10:55.882081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.882056 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.917202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.917168 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"kube-root-ca.crt\"" Oct 30 07:10:55.917202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.917201 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"openshift-service-ca.crt\"" Oct 30 07:10:55.917453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.917379 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"default-dockercfg-thbxc\"" Oct 30 07:10:55.917544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.917521 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a0b3f5\"/\"e2e-conflict-a-7c5e6b60-kserve-self-signed-certs\"" Oct 30 07:10:55.921655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.920367 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:10:55.944713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.944679 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:55.978170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978121 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.978374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978209 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsclv\" (UniqueName: \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.978436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978371 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.978487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.978544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978518 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:55.978606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:55.978573 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079574 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079669 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079707 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079756 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079858 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.080775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.079916 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsclv\" (UniqueName: \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.081214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.081170 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.081452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.081426 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.082327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.082299 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.083601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.083576 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.084793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.084769 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.106126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.106065 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsclv\" (UniqueName: \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\") pod \"e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.196842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.196804 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:56.500690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.500664 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:10:56.501658 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:56.501608 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f374e64_c108_4143_b92c_4d1d2cea56ae.slice/crio-9f70155554ffdf5b1367d9722d3309d5afbb06f285db954196f64f441117548e WatchSource:0}: Error finding container 9f70155554ffdf5b1367d9722d3309d5afbb06f285db954196f64f441117548e: Status 404 returned error can't find the container with id 9f70155554ffdf5b1367d9722d3309d5afbb06f285db954196f64f441117548e Oct 30 07:10:56.541152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:56.541121 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" event={"ID":"4f374e64-c108-4143-b92c-4d1d2cea56ae","Type":"ContainerStarted","Data":"9f70155554ffdf5b1367d9722d3309d5afbb06f285db954196f64f441117548e"} Oct 30 07:10:57.418536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:57.418504 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:57.549339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:57.549305 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" event={"ID":"4f374e64-c108-4143-b92c-4d1d2cea56ae","Type":"ContainerStarted","Data":"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64"} Oct 30 07:10:57.603577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:57.603503 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" podStartSLOduration=2.603480534 podStartE2EDuration="2.603480534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:10:57.602270297 +0000 UTC m=+821.293187737" watchObservedRunningTime="2026-10-30 07:10:57.603480534 +0000 UTC m=+821.294397974" Oct 30 07:10:58.197444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:58.197405 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:10:59.014810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.014775 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:10:59.140164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.140129 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s"] Oct 30 07:10:59.144178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.144153 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.163537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.163506 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s"] Oct 30 07:10:59.175212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.175181 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:10:59.175212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.175193 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:10:59.176079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.176058 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:10:59.176184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.176173 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:10:59.213912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.213874 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f695w\" (UniqueName: \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.214090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.213933 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.214090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.214078 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.315206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.315167 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f695w\" (UniqueName: \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.315425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.315219 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.315425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.315282 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.315998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.315974 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.326575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.326535 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.341261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.341228 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f695w\" (UniqueName: \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\") pod \"maas-api-revoke-sub-ca32fce9-gcx5s\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.459198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.459161 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:10:59.760156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:10:59.760127 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s"] Oct 30 07:10:59.761976 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:10:59.761947 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd214a166_1290_481a_b587_fb9a13208115.slice/crio-9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287 WatchSource:0}: Error finding container 9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287: Status 404 returned error can't find the container with id 9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287 Oct 30 07:11:00.512928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:00.512841 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:00.569693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:00.569626 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" event={"ID":"d214a166-1290-481a-b587-fb9a13208115","Type":"ContainerStarted","Data":"ed25c96c1faf6fb02a146bf2e277811c1efcb448ee225149fbce342c7f51d5a3"} Oct 30 07:11:00.569863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:00.569716 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" event={"ID":"d214a166-1290-481a-b587-fb9a13208115","Type":"ContainerStarted","Data":"9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287"} Oct 30 07:11:00.615941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:00.615889 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" podStartSLOduration=1.615876778 podStartE2EDuration="1.615876778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:00.614458383 +0000 UTC m=+824.305375820" watchObservedRunningTime="2026-10-30 07:11:00.615876778 +0000 UTC m=+824.306794214" Oct 30 07:11:01.576838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:01.576804 5894 generic.go:356] "Generic (PLEG): container finished" podID="d214a166-1290-481a-b587-fb9a13208115" containerID="ed25c96c1faf6fb02a146bf2e277811c1efcb448ee225149fbce342c7f51d5a3" exitCode=0 Oct 30 07:11:01.577298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:01.576887 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" event={"ID":"d214a166-1290-481a-b587-fb9a13208115","Type":"ContainerDied","Data":"ed25c96c1faf6fb02a146bf2e277811c1efcb448ee225149fbce342c7f51d5a3"} Oct 30 07:11:02.008491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.008406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:02.730337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.730310 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:11:02.855286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.855182 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\") pod \"d214a166-1290-481a-b587-fb9a13208115\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " Oct 30 07:11:02.855469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.855294 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\" (UniqueName: \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\") pod \"d214a166-1290-481a-b587-fb9a13208115\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " Oct 30 07:11:02.855469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.855385 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\") pod \"d214a166-1290-481a-b587-fb9a13208115\" (UID: \"d214a166-1290-481a-b587-fb9a13208115\") " Oct 30 07:11:02.855615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.855574 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca" pod "d214a166-1290-481a-b587-fb9a13208115" (UID: "d214a166-1290-481a-b587-fb9a13208115"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:11:02.855811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.855794 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d214a166-1290-481a-b587-fb9a13208115-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:02.857566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.857535 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w" pod "d214a166-1290-481a-b587-fb9a13208115" (UID: "d214a166-1290-481a-b587-fb9a13208115"). InnerVolumeSpecName "kube-api-access-f695w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:11:02.857744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.857565 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca" pod "d214a166-1290-481a-b587-fb9a13208115" (UID: "d214a166-1290-481a-b587-fb9a13208115"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:11:02.956476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.956435 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f695w\" (UniqueName: \"kubernetes.io/projected/d214a166-1290-481a-b587-fb9a13208115-kube-api-access-f695w\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:02.956476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:02.956475 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d214a166-1290-481a-b587-fb9a13208115-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:03.400825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:03.400793 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:11:03.517549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:03.517520 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:03.590177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:03.590146 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" Oct 30 07:11:03.590177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:03.590171 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s" event={"ID":"d214a166-1290-481a-b587-fb9a13208115","Type":"ContainerDied","Data":"9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287"} Oct 30 07:11:03.590455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:03.590219 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1aa941891609415b80b8b1aab953c66e1af12f9c4868f3d46bb7ee6c098287" Oct 30 07:11:04.829077 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:11:04.829038 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:11:05.240249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:05.240154 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:06.197477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:06.197440 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:11:06.211545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:06.211515 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:11:06.617876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:06.617848 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:11:06.814969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:06.814933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:08.321547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:08.321515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:09.461321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.461277 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs"] Oct 30 07:11:09.462032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.462010 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d214a166-1290-481a-b587-fb9a13208115" containerName="revoke-keys" Oct 30 07:11:09.462140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.462033 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d214a166-1290-481a-b587-fb9a13208115" containerName="revoke-keys" Oct 30 07:11:09.462184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.462175 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d214a166-1290-481a-b587-fb9a13208115" containerName="revoke-keys" Oct 30 07:11:09.466058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.466039 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.484227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.484196 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs"] Oct 30 07:11:09.496344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.496315 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:11:09.496505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.496354 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:11:09.496505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.496367 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:11:09.497029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.497002 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:11:09.627471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.627436 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzcs\" (UniqueName: \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.627686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.627508 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.627686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.627656 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.728590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.728491 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.728590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.728545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.728851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.728605 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzcs\" (UniqueName: \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.729288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.729258 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.730908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.730889 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.754374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.754344 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzcs\" (UniqueName: \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\") pod \"maas-api-revoke-sub-65f14b52-q2lgs\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.779331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.779301 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:09.875930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:09.875894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:10.083337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:10.083297 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs"] Oct 30 07:11:10.083697 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:11:10.083673 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f3bcb5_9d95_4804_aac3_c703a52fb50e.slice/crio-4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c WatchSource:0}: Error finding container 4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c: Status 404 returned error can't find the container with id 4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c Oct 30 07:11:10.631238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:10.631130 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" event={"ID":"96f3bcb5-9d95-4804-aac3-c703a52fb50e","Type":"ContainerStarted","Data":"d47bcd0ffb603fe7bee5aad7979c6d1c223669e3fd74e3c5cbab1084486a3ed9"} Oct 30 07:11:10.631238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:10.631204 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" event={"ID":"96f3bcb5-9d95-4804-aac3-c703a52fb50e","Type":"ContainerStarted","Data":"4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c"} Oct 30 07:11:10.678510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:10.678455 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" podStartSLOduration=1.678439133 podStartE2EDuration="1.678439133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:10.677596825 +0000 UTC m=+834.368514261" watchObservedRunningTime="2026-10-30 07:11:10.678439133 +0000 UTC m=+834.369356945" Oct 30 07:11:11.414457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:11.414422 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:11.639314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:11.639279 5894 generic.go:356] "Generic (PLEG): container finished" podID="96f3bcb5-9d95-4804-aac3-c703a52fb50e" containerID="d47bcd0ffb603fe7bee5aad7979c6d1c223669e3fd74e3c5cbab1084486a3ed9" exitCode=0 Oct 30 07:11:11.639733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:11.639324 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" event={"ID":"96f3bcb5-9d95-4804-aac3-c703a52fb50e","Type":"ContainerDied","Data":"d47bcd0ffb603fe7bee5aad7979c6d1c223669e3fd74e3c5cbab1084486a3ed9"} Oct 30 07:11:12.794192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.794164 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:12.935821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.935731 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:12.964810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.964763 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\" (UniqueName: \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\") pod \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " Oct 30 07:11:12.964993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.964944 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\") pod \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " Oct 30 07:11:12.964993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.964983 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\") pod \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\" (UID: \"96f3bcb5-9d95-4804-aac3-c703a52fb50e\") " Oct 30 07:11:12.965377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.965345 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca" pod "96f3bcb5-9d95-4804-aac3-c703a52fb50e" (UID: "96f3bcb5-9d95-4804-aac3-c703a52fb50e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:11:12.967237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.967209 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs" pod "96f3bcb5-9d95-4804-aac3-c703a52fb50e" (UID: "96f3bcb5-9d95-4804-aac3-c703a52fb50e"). InnerVolumeSpecName "kube-api-access-kgzcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:11:12.967508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:12.967479 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca" pod "96f3bcb5-9d95-4804-aac3-c703a52fb50e" (UID: "96f3bcb5-9d95-4804-aac3-c703a52fb50e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:11:13.066818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.066786 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgzcs\" (UniqueName: \"kubernetes.io/projected/96f3bcb5-9d95-4804-aac3-c703a52fb50e-kube-api-access-kgzcs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:13.066818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.066815 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:13.066818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.066826 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96f3bcb5-9d95-4804-aac3-c703a52fb50e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:11:13.652263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.652232 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" Oct 30 07:11:13.652263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.652260 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs" event={"ID":"96f3bcb5-9d95-4804-aac3-c703a52fb50e","Type":"ContainerDied","Data":"4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c"} Oct 30 07:11:13.652475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.652301 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ffb4e513002d5ba08ea8bdbb2419b263bbb29f8d4d3300b51d86cbf36aec24c" Oct 30 07:11:13.790051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.790016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:13.834675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.834629 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:11:13.835286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.835266 5894 cpu_manager.go:414] "containerMap: removing container" podUID="96f3bcb5-9d95-4804-aac3-c703a52fb50e" containerName="revoke-keys" Oct 30 07:11:13.835380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.835289 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f3bcb5-9d95-4804-aac3-c703a52fb50e" containerName="revoke-keys" Oct 30 07:11:13.835464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.835452 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f3bcb5-9d95-4804-aac3-c703a52fb50e" containerName="revoke-keys" Oct 30 07:11:13.838961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.838938 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.859746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.858668 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b72d79-openshift-default-dockercfg-m5v5m\"" Oct 30 07:11:13.873193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873148 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873195 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873232 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873332 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873463 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873521 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873602 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873678 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.873867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.873739 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254dz\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.875027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.874994 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:11:13.974517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974423 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974503 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974557 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974595 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974676 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254dz\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974732 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974779 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.974971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.974825 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975000 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975033 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975376 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975539 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975458 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.975770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.975508 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.977230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.977203 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:13.977394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:13.977374 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:14.000379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.000351 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:14.000696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.000672 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254dz\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:14.155190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.155152 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:14.474839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.474813 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:11:14.475796 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:11:14.475763 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7668dbc_9a31_4f5d_9c2b_317a5c024375.slice/crio-7de10756277e4e43ebd9fed4196283fbe0f3d0d961d767db64aa683a17166640 WatchSource:0}: Error finding container 7de10756277e4e43ebd9fed4196283fbe0f3d0d961d767db64aa683a17166640: Status 404 returned error can't find the container with id 7de10756277e4e43ebd9fed4196283fbe0f3d0d961d767db64aa683a17166640 Oct 30 07:11:14.478541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.478504 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:14.478686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.478578 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:14.478686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.478607 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:14.513336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.513309 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:14.673850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.673810 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" event={"ID":"e7668dbc-9a31-4f5d-9c2b-317a5c024375","Type":"ContainerStarted","Data":"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807"} Oct 30 07:11:14.674016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.673893 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" event={"ID":"e7668dbc-9a31-4f5d-9c2b-317a5c024375","Type":"ContainerStarted","Data":"7de10756277e4e43ebd9fed4196283fbe0f3d0d961d767db64aa683a17166640"} Oct 30 07:11:14.725971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:14.725857 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podStartSLOduration=1.725835564 podStartE2EDuration="1.725835564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:14.723117282 +0000 UTC m=+838.414034720" watchObservedRunningTime="2026-10-30 07:11:14.725835564 +0000 UTC m=+838.416753002" Oct 30 07:11:15.156016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:15.155981 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:15.158754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:15.158722 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:15.158924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:15.158788 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:16.069802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:16.069769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:16.156158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:16.156122 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:16.156667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:16.156191 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:16.515322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:16.515233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:17.155975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:17.155936 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:17.156159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:17.155998 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:17.839848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:17.839804 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:18.157194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:18.157089 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:18.157194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:18.157158 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:18.500555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:18.500456 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:19.156703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:19.156670 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:19.157124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:19.156724 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:19.532787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:19.532754 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:20.156528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:20.156488 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 07:11:20.156728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:20.156551 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 07:11:20.625065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:20.625031 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:11:21.187281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:21.187245 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:21.187884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:21.187866 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:21.188613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:21.188594 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:11:21.220284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:21.220250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:22.821624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:22.821591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:24.420655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:24.420579 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:24.542486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:24.542449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:25.678568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:25.678521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:26.086497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:26.086441 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:27.574042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:27.574003 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:29.138617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:29.138577 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:30.682440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:30.682398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:32.152883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:32.152845 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:33.654751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:33.654718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:34.831044 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:11:34.831005 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:11:35.209991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:35.209906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:36.727775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:36.727730 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:38.296091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:38.296059 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:39.019509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.019471 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 07:11:39.023775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.023752 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.042258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.042222 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 07:11:39.054737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.054695 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"kube-root-ca.crt\"" Oct 30 07:11:39.054737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.054712 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"openshift-service-ca.crt\"" Oct 30 07:11:39.054960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.054808 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"default-dockercfg-zzm9r\"" Oct 30 07:11:39.054960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.054852 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b72d79\"/\"distinct-w0-b72d79-kserve-self-signed-certs\"" Oct 30 07:11:39.074756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.074722 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:39.162996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.162962 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.163207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.163006 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.163279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.163197 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.163279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.163251 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxsbj\" (UniqueName: \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.163388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.163350 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.163388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.163377 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.264875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.264821 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.264889 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.264990 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.265030 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxsbj\" (UniqueName: \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.265102 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.265131 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.265877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.265851 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.266008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.265937 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.266073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.266049 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.268060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.268032 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.268196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.268175 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.291812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.291778 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxsbj\" (UniqueName: \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\") pod \"distinct-w0-b72d79-kserve-588f6f7948-tgb56\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.337402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.337369 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:39.650500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.650472 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 07:11:39.651035 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:11:39.651005 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e8789b2_aaff_4d00_8c4b_66af9c2cfb00.slice/crio-9ecfc87092b75c5fa95f3f1c24b345aea8dfc2f6a8e3dead6d7f494d2473fcdd WatchSource:0}: Error finding container 9ecfc87092b75c5fa95f3f1c24b345aea8dfc2f6a8e3dead6d7f494d2473fcdd: Status 404 returned error can't find the container with id 9ecfc87092b75c5fa95f3f1c24b345aea8dfc2f6a8e3dead6d7f494d2473fcdd Oct 30 07:11:39.833543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.833506 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:39.841057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.841019 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" event={"ID":"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00","Type":"ContainerStarted","Data":"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca"} Oct 30 07:11:39.841262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.841073 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" event={"ID":"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00","Type":"ContainerStarted","Data":"9ecfc87092b75c5fa95f3f1c24b345aea8dfc2f6a8e3dead6d7f494d2473fcdd"} Oct 30 07:11:39.889743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:39.889686 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podStartSLOduration=1.8896730210000001 podStartE2EDuration="1.889673021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:39.888452853 +0000 UTC m=+863.579370290" watchObservedRunningTime="2026-10-30 07:11:39.889673021 +0000 UTC m=+863.580590458" Oct 30 07:11:40.338089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:40.338049 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:41.413065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:41.413025 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:42.111168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:42.111132 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:11:43.032557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:43.032496 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:44.137493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.137452 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:11:44.142027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.142003 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.160109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.160076 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e5fa2d-openshift-default-dockercfg-sppbj\"" Oct 30 07:11:44.177785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.177752 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:11:44.222055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222019 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222071 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222165 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222202 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222223 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222239 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222277 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ktr2\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222398 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.222465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.222451 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323021 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323102 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323138 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323174 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323230 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323275 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323306 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323334 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ktr2\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323380 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323463 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323526 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323826 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.323917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.323900 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.324309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.324286 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.325691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.325667 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.325964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.325945 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.349216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.349181 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.349461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.349439 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ktr2\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.458355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.458243 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:44.608819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.608789 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:44.781041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.781008 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:11:44.781419 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:11:44.781391 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55276e15_4d9c_42dd_ae4d_95859b544e27.slice/crio-d616b10b406cf0169af76e0d9d71fce2540a1095d87d946792b283c3ee9de2a9 WatchSource:0}: Error finding container d616b10b406cf0169af76e0d9d71fce2540a1095d87d946792b283c3ee9de2a9: Status 404 returned error can't find the container with id d616b10b406cf0169af76e0d9d71fce2540a1095d87d946792b283c3ee9de2a9 Oct 30 07:11:44.784219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.784184 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:44.784298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.784265 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:44.784337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.784309 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:11:44.877499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.877463 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" event={"ID":"55276e15-4d9c-42dd-ae4d-95859b544e27","Type":"ContainerStarted","Data":"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1"} Oct 30 07:11:44.877700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.877515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" event={"ID":"55276e15-4d9c-42dd-ae4d-95859b544e27","Type":"ContainerStarted","Data":"d616b10b406cf0169af76e0d9d71fce2540a1095d87d946792b283c3ee9de2a9"} Oct 30 07:11:44.927645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:44.927559 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podStartSLOduration=0.927539246 podStartE2EDuration="927.539246ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:44.927013537 +0000 UTC m=+868.617930967" watchObservedRunningTime="2026-10-30 07:11:44.927539246 +0000 UTC m=+868.618456683" Oct 30 07:11:45.458844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:45.458804 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:45.460556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:45.460531 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:45.460691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:45.460580 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:46.238542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:46.238503 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:46.458607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:46.458572 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:46.458800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:46.458648 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:47.459501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:47.459457 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:47.460043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:47.459549 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:47.924366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:47.924325 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:48.459648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.459609 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:48.460051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.459697 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:48.903504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.903458 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 07:11:48.907798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.907773 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 07:11:48.907960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.907866 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.941777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.941742 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e5fa2d\"/\"distinct-2-w2-e5fa2d-kserve-self-signed-certs\"" Oct 30 07:11:48.950400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.950365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998228 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998322 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998361 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqj6\" (UniqueName: \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998471 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:48.998659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:48.998497 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112613 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqj6\" (UniqueName: \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112728 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112761 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112786 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112899 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.112962 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.113660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.113421 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.120662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.114785 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.120662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.115026 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.126779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.123569 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.128113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.128090 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.140451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.140415 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqj6\" (UniqueName: \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\") pod \"distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.224745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.224658 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:49.339673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.339188 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:49.355015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.354697 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:49.458596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.458560 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:49.458818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.458620 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:49.570328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.570301 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 07:11:49.691664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.690656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:49.936460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.936363 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" event={"ID":"6ebe0fb1-3f63-4b93-b602-833f29138e5c","Type":"ContainerStarted","Data":"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9"} Oct 30 07:11:49.936460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.936413 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" event={"ID":"6ebe0fb1-3f63-4b93-b602-833f29138e5c","Type":"ContainerStarted","Data":"9d762eaffe6b62886ca08eec254d466f94333b31f73ae542b0cf346d7b625fde"} Oct 30 07:11:49.948087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:49.948056 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 07:11:50.043438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:50.043389 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" podStartSLOduration=2.043374637 podStartE2EDuration="2.043374637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:11:49.995902653 +0000 UTC m=+873.686820090" watchObservedRunningTime="2026-10-30 07:11:50.043374637 +0000 UTC m=+873.734292126" Oct 30 07:11:50.225122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:50.225035 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:50.459567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:50.459526 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 07:11:50.459746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:50.459590 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 07:11:51.286515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:51.286476 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:51.743959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:51.743870 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:51.744304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:51.744276 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:51.745431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:51.745399 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:11:52.907802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:52.907762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:54.420628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:54.420587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:55.901465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:55.901434 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:57.338021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:57.337978 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:58.902372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:58.902305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:11:59.038022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:59.037981 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:11:59.225173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:59.225087 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:11:59.237799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:11:59.237772 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:12:00.016372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:00.016341 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 07:12:00.496115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:00.496077 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:00.624981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:00.624944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:01.280877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:01.280830 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:02.119326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:02.119293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:03.610227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:03.610165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:04.151137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.151098 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:12:04.155294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.155268 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.172195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.172170 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a0b3f5-openshift-default-dockercfg-kzcpc\"" Oct 30 07:12:04.188614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.188580 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:12:04.299207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299162 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299273 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmbg7\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299317 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299432 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299460 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299505 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299552 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299665 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.299854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.299833 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401044 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401086 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401124 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401163 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401201 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401346 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401370 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmbg7\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401408 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401725 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.401797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401777 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.402008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.401982 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.402119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.402054 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.402268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.402236 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.403719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.403699 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.403874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.403857 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.427694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.427661 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.427867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.427774 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmbg7\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.469935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.469897 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:04.800068 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:04.800024 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda06143b_f46e_4b7e_a6f2_e71ba6a1fad0.slice/crio-ad11a20e18ade2fd23bdb8572f155f1d47886f4dd83a3c8f76529917e7b9d135 WatchSource:0}: Error finding container ad11a20e18ade2fd23bdb8572f155f1d47886f4dd83a3c8f76529917e7b9d135: Status 404 returned error can't find the container with id ad11a20e18ade2fd23bdb8572f155f1d47886f4dd83a3c8f76529917e7b9d135 Oct 30 07:12:04.800537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.800194 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:12:04.803018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.802986 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:04.803106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.803053 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:04.803106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:04.803083 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:04.835989 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:12:04.835955 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:12:05.035443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.035408 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" event={"ID":"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0","Type":"ContainerStarted","Data":"cad9f4ad5500e093a9fe19ac80a439f335b2f720349f4f7b4fedf562aa20aece"} Oct 30 07:12:05.035623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.035469 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" event={"ID":"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0","Type":"ContainerStarted","Data":"ad11a20e18ade2fd23bdb8572f155f1d47886f4dd83a3c8f76529917e7b9d135"} Oct 30 07:12:05.086671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.086523 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podStartSLOduration=1.086506499 podStartE2EDuration="1.086506499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:05.085250945 +0000 UTC m=+888.776168384" watchObservedRunningTime="2026-10-30 07:12:05.086506499 +0000 UTC m=+888.777423940" Oct 30 07:12:05.114402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.114260 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:05.470921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.470842 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:05.472089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.472065 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:05.472195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:05.472113 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:06.470429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:06.470389 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:06.470990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:06.470453 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:06.629700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:06.629661 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:07.481103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:07.481060 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:07.481708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:07.481674 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:07.677668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:07.672308 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:12:07.710137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:07.710106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:08.439230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:08.439192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:08.481667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:08.480010 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:08.481667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:08.480093 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:09.495824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:09.495786 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:09.496304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:09.496266 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:10.148231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:10.148195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:10.470340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:10.470258 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:10.470340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:10.470308 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:11.470479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:11.470441 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 07:12:11.470924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:11.470525 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 07:12:11.796427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:11.796391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:13.343752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.343717 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:13.344454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.344386 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:13.371214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.371182 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:12:13.387453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.387414 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:13.935212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.935175 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:12:13.939530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.939503 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:13.975876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:13.975828 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:12:14.021745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.021703 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.021950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.021810 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.021950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.021848 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.021950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.021909 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.022127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.022044 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.022185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.022162 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.022243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.022185 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.022243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.022225 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5pw2\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.022345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.022265 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.123899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.123864 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.123899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.123899 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.123926 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5pw2\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.123948 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.123973 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124013 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124039 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124087 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124115 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124403 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124673 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124742 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124761 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.124893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.124862 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.126379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.126350 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.126474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.126465 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.149359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.149331 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.149628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.149602 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5pw2\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.255932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.255832 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:14.609999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.609960 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:12:14.610761 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:14.610720 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036ce68f_66d2_4260_aeb9_a60084fb2e10.slice/crio-d5ae5d0e37c1f05426a7816dbe2f8eaf6987ce2ed43c31ebb8d4e2b8020f86b3 WatchSource:0}: Error finding container d5ae5d0e37c1f05426a7816dbe2f8eaf6987ce2ed43c31ebb8d4e2b8020f86b3: Status 404 returned error can't find the container with id d5ae5d0e37c1f05426a7816dbe2f8eaf6987ce2ed43c31ebb8d4e2b8020f86b3 Oct 30 07:12:14.613232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.613196 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:14.613338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.613278 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:14.613338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:14.613325 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:15.003218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.003123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:15.126577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.126519 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" event={"ID":"036ce68f-66d2-4260-aeb9-a60084fb2e10","Type":"ContainerStarted","Data":"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e"} Oct 30 07:12:15.126783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.126594 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" event={"ID":"036ce68f-66d2-4260-aeb9-a60084fb2e10","Type":"ContainerStarted","Data":"d5ae5d0e37c1f05426a7816dbe2f8eaf6987ce2ed43c31ebb8d4e2b8020f86b3"} Oct 30 07:12:15.256761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.256672 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:15.258175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.258147 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:15.258308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:15.258202 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:16.256472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:16.256438 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:16.256891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:16.256493 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:16.728577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:16.728539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:17.256964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:17.256930 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:17.257388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:17.256986 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:18.257229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:18.257188 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:18.257689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:18.257258 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:18.327856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:18.327821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:19.257146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:19.257113 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:19.257312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:19.257168 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:19.952220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:19.952172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:20.107412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:20.107367 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:20.257939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:20.257839 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 07:12:20.257939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:20.257910 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 07:12:21.065711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.065660 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podStartSLOduration=8.065618365 podStartE2EDuration="8.065618365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:15.191961095 +0000 UTC m=+898.882878534" watchObservedRunningTime="2026-10-30 07:12:21.065618365 +0000 UTC m=+904.756535802" Oct 30 07:12:21.065905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.065880 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z"] Oct 30 07:12:21.070990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.070967 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.090001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.089969 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z"] Oct 30 07:12:21.100800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.100767 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:21.100956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.100823 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:21.102234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.102205 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:21.102408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.102316 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:21.234246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.234114 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.234246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.234198 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.234515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.234385 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtc2r\" (UniqueName: \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.335233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.335152 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtc2r\" (UniqueName: \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.335233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.335214 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.335653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.335251 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.336160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.336129 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.337573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.337551 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.364304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.364268 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtc2r\" (UniqueName: \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\") pod \"maas-api-revoke-sub-bbf139c6-lph5z\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.385099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.385067 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:21.498456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.498407 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:12:21.537497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.537466 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:21.537902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.537876 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:21.539574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.539553 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:12:21.634538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.634452 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:21.698867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:21.698834 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z"] Oct 30 07:12:21.699979 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:21.699949 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2434627b_8a78_4863_a218_9c4f6363fc78.slice/crio-5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7 WatchSource:0}: Error finding container 5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7: Status 404 returned error can't find the container with id 5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7 Oct 30 07:12:22.165449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.165359 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:12:22.170281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.170252 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.173805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.173771 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" event={"ID":"2434627b-8a78-4863-a218-9c4f6363fc78","Type":"ContainerStarted","Data":"d093e364d908c9b3b22feceaec34d15a80d5767f3180fa900906cdf050cda5ce"} Oct 30 07:12:22.173964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.173828 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" event={"ID":"2434627b-8a78-4863-a218-9c4f6363fc78","Type":"ContainerStarted","Data":"5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7"} Oct 30 07:12:22.189294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.189265 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 07:12:22.192239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.192212 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:12:22.263909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.263854 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" podStartSLOduration=1.263836223 podStartE2EDuration="1.263836223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:22.262562169 +0000 UTC m=+905.953479606" watchObservedRunningTime="2026-10-30 07:12:22.263836223 +0000 UTC m=+905.954753698" Oct 30 07:12:22.347256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347214 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347269 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347298 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347494 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347709 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347787 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.347919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347829 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.348080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.347922 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-586kf\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449238 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-586kf\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449331 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449353 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449374 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449389 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449409 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449497 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449547 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449590 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.449966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449925 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.450021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.449991 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.450082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.450035 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.450148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.450079 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.450309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.450287 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.451790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.451768 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.452217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.452196 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.475191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.475158 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-586kf\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.475542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.475521 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.485559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.485528 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:22.723759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.723680 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" podUID="4f374e64-c108-4143-b92c-4d1d2cea56ae" containerName="main" containerID="cri-o://a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64" gracePeriod=45 Oct 30 07:12:22.811361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.811334 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:12:22.816699 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:22.816669 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112f2bce_f857_46b6_acc7_293529163026.slice/crio-917dad6c82a5fdca6ea8092ff6fa961eca0265288286aabeba5f205713f84329 WatchSource:0}: Error finding container 917dad6c82a5fdca6ea8092ff6fa961eca0265288286aabeba5f205713f84329: Status 404 returned error can't find the container with id 917dad6c82a5fdca6ea8092ff6fa961eca0265288286aabeba5f205713f84329 Oct 30 07:12:22.829199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.829155 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.829284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.829251 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.829319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.829300 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:22.995717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:22.995691 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:12:23.157469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157437 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157489 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157521 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157689 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\" (UniqueName: \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157727 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157763 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\") pod \"4f374e64-c108-4143-b92c-4d1d2cea56ae\" (UID: \"4f374e64-c108-4143-b92c-4d1d2cea56ae\") " Oct 30 07:12:23.157945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157786 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:23.157945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157798 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:23.157945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.157868 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:23.158220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.158193 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.158261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.158220 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.158261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.158235 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.160252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.160228 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:23.160252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.160237 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:23.160433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.160368 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv" pod "4f374e64-c108-4143-b92c-4d1d2cea56ae" (UID: "4f374e64-c108-4143-b92c-4d1d2cea56ae"). InnerVolumeSpecName "kube-api-access-wsclv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:23.182383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.182353 5894 generic.go:356] "Generic (PLEG): container finished" podID="2434627b-8a78-4863-a218-9c4f6363fc78" containerID="d093e364d908c9b3b22feceaec34d15a80d5767f3180fa900906cdf050cda5ce" exitCode=0 Oct 30 07:12:23.182522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.182435 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" event={"ID":"2434627b-8a78-4863-a218-9c4f6363fc78","Type":"ContainerDied","Data":"d093e364d908c9b3b22feceaec34d15a80d5767f3180fa900906cdf050cda5ce"} Oct 30 07:12:23.184326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.184300 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" event={"ID":"112f2bce-f857-46b6-acc7-293529163026","Type":"ContainerStarted","Data":"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d"} Oct 30 07:12:23.184472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.184350 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" event={"ID":"112f2bce-f857-46b6-acc7-293529163026","Type":"ContainerStarted","Data":"917dad6c82a5fdca6ea8092ff6fa961eca0265288286aabeba5f205713f84329"} Oct 30 07:12:23.185614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.185595 5894 generic.go:356] "Generic (PLEG): container finished" podID="4f374e64-c108-4143-b92c-4d1d2cea56ae" containerID="a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64" exitCode=0 Oct 30 07:12:23.185709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.185645 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" Oct 30 07:12:23.185709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.185689 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" event={"ID":"4f374e64-c108-4143-b92c-4d1d2cea56ae","Type":"ContainerDied","Data":"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64"} Oct 30 07:12:23.185780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.185719 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4" event={"ID":"4f374e64-c108-4143-b92c-4d1d2cea56ae","Type":"ContainerDied","Data":"9f70155554ffdf5b1367d9722d3309d5afbb06f285db954196f64f441117548e"} Oct 30 07:12:23.185780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.185736 5894 scope.go:122] "RemoveContainer" containerID="a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64" Oct 30 07:12:23.195779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.195731 5894 scope.go:122] "RemoveContainer" containerID="a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64" Oct 30 07:12:23.196052 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:12:23.196034 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64\": container with ID starting with a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64 not found: ID does not exist" containerID="a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64" Oct 30 07:12:23.196096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.196062 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64"} err="failed to get container status \"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64\": rpc error: code = NotFound desc = could not find container \"a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64\": container with ID starting with a31d76e1b799ebac86026d6fab4c819a85d3f46b0f72eafa55ba4ac81381ca64 not found: ID does not exist" Oct 30 07:12:23.202988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.202944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:23.259392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.259307 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsclv\" (UniqueName: \"kubernetes.io/projected/4f374e64-c108-4143-b92c-4d1d2cea56ae-kube-api-access-wsclv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.259392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.259335 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f374e64-c108-4143-b92c-4d1d2cea56ae-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.259392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.259351 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f374e64-c108-4143-b92c-4d1d2cea56ae-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:23.271178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.271148 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:12:23.274603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.274575 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a0b3f5/e2e-conflict-a-7c5e6b60-kserve-85dc98d997-s6zp4"] Oct 30 07:12:23.323491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.323403 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podStartSLOduration=1.323381973 podStartE2EDuration="1.323381973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:23.320025666 +0000 UTC m=+907.010943104" watchObservedRunningTime="2026-10-30 07:12:23.323381973 +0000 UTC m=+907.014299408" Oct 30 07:12:23.488599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.487031 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:23.495827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.495790 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:23.496007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:23.495857 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:24.367793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.367766 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:24.431881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.431848 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f374e64-c108-4143-b92c-4d1d2cea56ae" path="/var/lib/kubelet/pods/4f374e64-c108-4143-b92c-4d1d2cea56ae/volumes" Oct 30 07:12:24.496108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.496022 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:24.496108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.496095 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:24.498342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.498316 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\") pod \"2434627b-8a78-4863-a218-9c4f6363fc78\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " Oct 30 07:12:24.498492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.498479 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\") pod \"2434627b-8a78-4863-a218-9c4f6363fc78\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " Oct 30 07:12:24.498575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.498523 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\" (UniqueName: \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\") pod \"2434627b-8a78-4863-a218-9c4f6363fc78\" (UID: \"2434627b-8a78-4863-a218-9c4f6363fc78\") " Oct 30 07:12:24.498978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.498944 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca" pod "2434627b-8a78-4863-a218-9c4f6363fc78" (UID: "2434627b-8a78-4863-a218-9c4f6363fc78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:24.500791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.500767 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca" pod "2434627b-8a78-4863-a218-9c4f6363fc78" (UID: "2434627b-8a78-4863-a218-9c4f6363fc78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:24.500885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.500779 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r" pod "2434627b-8a78-4863-a218-9c4f6363fc78" (UID: "2434627b-8a78-4863-a218-9c4f6363fc78"). InnerVolumeSpecName "kube-api-access-qtc2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:24.600567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.600525 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:24.600567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.600564 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtc2r\" (UniqueName: \"kubernetes.io/projected/2434627b-8a78-4863-a218-9c4f6363fc78-kube-api-access-qtc2r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:24.601008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.600579 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2434627b-8a78-4863-a218-9c4f6363fc78-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:24.934470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:24.934436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:25.209404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.209306 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" Oct 30 07:12:25.209404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.209320 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z" event={"ID":"2434627b-8a78-4863-a218-9c4f6363fc78","Type":"ContainerDied","Data":"5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7"} Oct 30 07:12:25.209404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.209368 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f33346fb8aafb8b9a1f6ffb5a593824b657f05afe2f66b4e3376f13545c8ab7" Oct 30 07:12:25.492715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.492024 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:25.492715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.492099 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:25.822361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:25.822326 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:26.486069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:26.486032 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:26.486375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:26.486342 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:26.541792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:26.541757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:27.029320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:27.029156 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:27.489132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:27.489041 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:27.489373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:27.489345 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:28.330959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:28.330924 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:28.493096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:28.493064 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:28.493372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:28.493346 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:29.295912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.295876 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 07:12:29.296780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296759 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2434627b-8a78-4863-a218-9c4f6363fc78" containerName="revoke-keys" Oct 30 07:12:29.296852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296783 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2434627b-8a78-4863-a218-9c4f6363fc78" containerName="revoke-keys" Oct 30 07:12:29.296852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296842 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4f374e64-c108-4143-b92c-4d1d2cea56ae" containerName="main" Oct 30 07:12:29.296945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296851 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f374e64-c108-4143-b92c-4d1d2cea56ae" containerName="main" Oct 30 07:12:29.297006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296965 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f374e64-c108-4143-b92c-4d1d2cea56ae" containerName="main" Oct 30 07:12:29.297006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.296988 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2434627b-8a78-4863-a218-9c4f6363fc78" containerName="revoke-keys" Oct 30 07:12:29.302121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.302099 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.323865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.323829 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 07:12:29.360855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.360824 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.360880 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.360959 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361014 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361035 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361122 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgbmf\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361180 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.361263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.361237 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.462780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462212 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.462780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462354 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.462780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462429 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.462780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462513 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.462780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462610 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462895 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462954 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.462983 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463102 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463112 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgbmf\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463126 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463105 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463316 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.463672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.463610 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.465541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.465512 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.465710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.465664 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.486181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.486151 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 07:12:29.486329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.486205 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 07:12:29.487933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.487907 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.488154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.488131 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgbmf\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.621565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.621471 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:29.955763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.955666 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 07:12:29.960423 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:29.960390 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff106477_2682_4035_bfda_adffc03ed0c7.slice/crio-eebb4a23b0c82a0e27a0aee7bde6d39806334e27491d4f46ab41c66f28e28b41 WatchSource:0}: Error finding container eebb4a23b0c82a0e27a0aee7bde6d39806334e27491d4f46ab41c66f28e28b41: Status 404 returned error can't find the container with id eebb4a23b0c82a0e27a0aee7bde6d39806334e27491d4f46ab41c66f28e28b41 Oct 30 07:12:29.964721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.964629 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:29.964845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.964740 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:29.964845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:29.964785 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:30.032394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.032354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:30.181393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.181355 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw"] Oct 30 07:12:30.185726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.185701 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.206699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.205062 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:30.206699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.205125 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:30.206699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.205627 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:30.206699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.205744 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:30.206699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.206513 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw"] Oct 30 07:12:30.260394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.260213 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" event={"ID":"ff106477-2682-4035-bfda-adffc03ed0c7","Type":"ContainerStarted","Data":"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144"} Oct 30 07:12:30.260394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.260272 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" event={"ID":"ff106477-2682-4035-bfda-adffc03ed0c7","Type":"ContainerStarted","Data":"eebb4a23b0c82a0e27a0aee7bde6d39806334e27491d4f46ab41c66f28e28b41"} Oct 30 07:12:30.273179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.273138 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.273362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.273293 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmq9\" (UniqueName: \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.273512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.273489 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.310371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.310309 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podStartSLOduration=1.310291477 podStartE2EDuration="1.310291477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:30.309730954 +0000 UTC m=+914.000648392" watchObservedRunningTime="2026-10-30 07:12:30.310291477 +0000 UTC m=+914.001208915" Oct 30 07:12:30.374730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.374689 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.375170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.374783 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.375170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.374903 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmq9\" (UniqueName: \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.375702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.375670 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.377290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.377271 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.401135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.401109 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmq9\" (UniqueName: \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\") pod \"maas-api-revoke-sub-5a8efd01-dxffw\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.490874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.490800 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:30.491283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.491258 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:30.492030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.492012 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:12:30.503224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.503199 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:30.621945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.621913 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:30.813284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:30.813257 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw"] Oct 30 07:12:30.813574 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:30.813550 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11947bc_970f_46d5_b37b_b8b10746dac2.slice/crio-53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284 WatchSource:0}: Error finding container 53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284: Status 404 returned error can't find the container with id 53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284 Oct 30 07:12:31.268124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.268078 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" event={"ID":"a11947bc-970f-46d5-b37b-b8b10746dac2","Type":"ContainerStarted","Data":"38a7de25b3cbed389f42a7d6c449ce35a8e06266933c30746a6953abe7cd6b22"} Oct 30 07:12:31.268334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.268140 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" event={"ID":"a11947bc-970f-46d5-b37b-b8b10746dac2","Type":"ContainerStarted","Data":"53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284"} Oct 30 07:12:31.315840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.315785 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" podStartSLOduration=1.315768817 podStartE2EDuration="1.315768817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:31.31377305 +0000 UTC m=+915.004690487" watchObservedRunningTime="2026-10-30 07:12:31.315768817 +0000 UTC m=+915.006686271" Oct 30 07:12:31.629840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.629805 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:31.630546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.630490 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:31.905749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:31.896021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:32.428659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:32.427726 5894 generic.go:356] "Generic (PLEG): container finished" podID="a11947bc-970f-46d5-b37b-b8b10746dac2" containerID="38a7de25b3cbed389f42a7d6c449ce35a8e06266933c30746a6953abe7cd6b22" exitCode=0 Oct 30 07:12:32.432657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:32.428868 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" event={"ID":"a11947bc-970f-46d5-b37b-b8b10746dac2","Type":"ContainerDied","Data":"38a7de25b3cbed389f42a7d6c449ce35a8e06266933c30746a6953abe7cd6b22"} Oct 30 07:12:32.630654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:32.630288 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:32.630654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:32.630381 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:33.622222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.622193 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:33.622359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.622250 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:33.631538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.631509 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:33.712252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.710117 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\") pod \"a11947bc-970f-46d5-b37b-b8b10746dac2\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " Oct 30 07:12:33.712252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.710335 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\" (UniqueName: \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\") pod \"a11947bc-970f-46d5-b37b-b8b10746dac2\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " Oct 30 07:12:33.712252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.710493 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\") pod \"a11947bc-970f-46d5-b37b-b8b10746dac2\" (UID: \"a11947bc-970f-46d5-b37b-b8b10746dac2\") " Oct 30 07:12:33.712252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.711476 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca" pod "a11947bc-970f-46d5-b37b-b8b10746dac2" (UID: "a11947bc-970f-46d5-b37b-b8b10746dac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:33.716394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.715210 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9" pod "a11947bc-970f-46d5-b37b-b8b10746dac2" (UID: "a11947bc-970f-46d5-b37b-b8b10746dac2"). InnerVolumeSpecName "kube-api-access-zxmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:33.716394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.716314 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca" pod "a11947bc-970f-46d5-b37b-b8b10746dac2" (UID: "a11947bc-970f-46d5-b37b-b8b10746dac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:33.812175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.812144 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxmq9\" (UniqueName: \"kubernetes.io/projected/a11947bc-970f-46d5-b37b-b8b10746dac2-kube-api-access-zxmq9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:33.812175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.812172 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:33.812175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.812182 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a11947bc-970f-46d5-b37b-b8b10746dac2-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:33.957966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:33.957936 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:34.442384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:34.442351 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" event={"ID":"a11947bc-970f-46d5-b37b-b8b10746dac2","Type":"ContainerDied","Data":"53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284"} Oct 30 07:12:34.442517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:34.442391 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw" Oct 30 07:12:34.442517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:34.442409 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53e778a0f085fbca7e46cb8912fd6dd7df1c61bb9d474ac937c4e1fde10b4284" Oct 30 07:12:34.622019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:34.621985 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:34.622196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:34.622040 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:34.832124 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:12:34.832087 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:12:35.550155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:35.550116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:35.622263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:35.622233 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:35.622440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:35.622282 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:36.178657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.173532 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn"] Oct 30 07:12:36.187661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.184457 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a11947bc-970f-46d5-b37b-b8b10746dac2" containerName="revoke-keys" Oct 30 07:12:36.187661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.184492 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11947bc-970f-46d5-b37b-b8b10746dac2" containerName="revoke-keys" Oct 30 07:12:36.187661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.184672 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11947bc-970f-46d5-b37b-b8b10746dac2" containerName="revoke-keys" Oct 30 07:12:36.199224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.199195 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.204328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.204296 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn"] Oct 30 07:12:36.216442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.216417 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:36.230648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.230585 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:36.230648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.230587 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:36.230822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.230769 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:36.340564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.340529 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.340765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.340661 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.340765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.340708 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9mxl\" (UniqueName: \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.441569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.441495 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.441746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.441577 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.441746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.441617 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9mxl\" (UniqueName: \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.442235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.442208 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.443901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.443876 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.467713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.467685 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9mxl\" (UniqueName: \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\") pod \"maas-api-revoke-sub-060d7f6e-tgzwn\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.520592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.520548 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:36.622989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.622624 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 07:12:36.622989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.622694 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 07:12:36.833269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:36.833240 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn"] Oct 30 07:12:36.836875 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:36.836837 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c652383_ffbd_404b_8887_4a4eac003599.slice/crio-dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd WatchSource:0}: Error finding container dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd: Status 404 returned error can't find the container with id dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd Oct 30 07:12:37.216486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:37.216429 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:37.468410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:37.468319 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" event={"ID":"2c652383-ffbd-404b-8887-4a4eac003599","Type":"ContainerStarted","Data":"83727b111506ce56c22c6c330a14deb9a79f152930f21eb17781155a394fd703"} Oct 30 07:12:37.468410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:37.468386 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" event={"ID":"2c652383-ffbd-404b-8887-4a4eac003599","Type":"ContainerStarted","Data":"dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd"} Oct 30 07:12:37.520049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:37.519988 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" podStartSLOduration=1.519970099 podStartE2EDuration="1.519970099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:37.517850552 +0000 UTC m=+921.208767988" watchObservedRunningTime="2026-10-30 07:12:37.519970099 +0000 UTC m=+921.210887538" Oct 30 07:12:38.476465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.476424 5894 generic.go:356] "Generic (PLEG): container finished" podID="2c652383-ffbd-404b-8887-4a4eac003599" containerID="83727b111506ce56c22c6c330a14deb9a79f152930f21eb17781155a394fd703" exitCode=0 Oct 30 07:12:38.476855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.476565 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" event={"ID":"2c652383-ffbd-404b-8887-4a4eac003599","Type":"ContainerDied","Data":"83727b111506ce56c22c6c330a14deb9a79f152930f21eb17781155a394fd703"} Oct 30 07:12:38.663825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.663784 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:12:38.664002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.663854 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:12:38.963125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.963072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:38.989410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.989381 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:38.989746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.989719 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:38.990624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:38.990604 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 07:12:39.634673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.634645 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:39.785578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.785548 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\") pod \"2c652383-ffbd-404b-8887-4a4eac003599\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " Oct 30 07:12:39.785749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.785692 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\") pod \"2c652383-ffbd-404b-8887-4a4eac003599\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " Oct 30 07:12:39.785749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.785728 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\" (UniqueName: \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\") pod \"2c652383-ffbd-404b-8887-4a4eac003599\" (UID: \"2c652383-ffbd-404b-8887-4a4eac003599\") " Oct 30 07:12:39.786118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.786087 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca" pod "2c652383-ffbd-404b-8887-4a4eac003599" (UID: "2c652383-ffbd-404b-8887-4a4eac003599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:39.786239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.786173 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:39.787702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.787672 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca" pod "2c652383-ffbd-404b-8887-4a4eac003599" (UID: "2c652383-ffbd-404b-8887-4a4eac003599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:39.787802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.787773 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl" pod "2c652383-ffbd-404b-8887-4a4eac003599" (UID: "2c652383-ffbd-404b-8887-4a4eac003599"). InnerVolumeSpecName "kube-api-access-v9mxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:39.887371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.887336 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9mxl\" (UniqueName: \"kubernetes.io/projected/2c652383-ffbd-404b-8887-4a4eac003599-kube-api-access-v9mxl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:39.887371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:39.887368 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c652383-ffbd-404b-8887-4a4eac003599-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:40.339188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.339147 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:12:40.491465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.491438 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" Oct 30 07:12:40.491682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.491475 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn" event={"ID":"2c652383-ffbd-404b-8887-4a4eac003599","Type":"ContainerDied","Data":"dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd"} Oct 30 07:12:40.491682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.491536 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc38e6548cf7706959cec220c9221859562b95facfa114a5d6ab183dd49ba3bd" Oct 30 07:12:40.577559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.577512 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:40.693040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.692960 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p"] Oct 30 07:12:40.693488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.693474 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2c652383-ffbd-404b-8887-4a4eac003599" containerName="revoke-keys" Oct 30 07:12:40.693531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.693489 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c652383-ffbd-404b-8887-4a4eac003599" containerName="revoke-keys" Oct 30 07:12:40.693613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.693604 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c652383-ffbd-404b-8887-4a4eac003599" containerName="revoke-keys" Oct 30 07:12:40.697289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.697266 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.716359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.714817 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:40.716359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.714884 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:40.716359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.716237 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p"] Oct 30 07:12:40.727274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.727249 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:40.727466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.727286 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:40.800340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.800304 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.800514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.800481 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.800514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.800508 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbc2\" (UniqueName: \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.901815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.901775 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.901998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.901897 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.901998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.901925 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbc2\" (UniqueName: \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.902619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.902595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.904342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.904318 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:40.928028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:40.927992 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbc2\" (UniqueName: \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\") pod \"maas-api-revoke-sub-33f58211-r8d6p\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:41.014756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:41.014553 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:41.376007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:41.375973 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p"] Oct 30 07:12:41.381044 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:41.381010 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c64eea_e874_4ae2_a6db_61f394284ee3.slice/crio-101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9 WatchSource:0}: Error finding container 101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9: Status 404 returned error can't find the container with id 101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9 Oct 30 07:12:41.509149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:41.509115 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" event={"ID":"b0c64eea-e874-4ae2-a6db-61f394284ee3","Type":"ContainerStarted","Data":"101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9"} Oct 30 07:12:41.738044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:41.737949 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:12:41.738838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:41.738800 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" containerID="cri-o://78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39" gracePeriod=30 Oct 30 07:12:42.527659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:42.527390 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" event={"ID":"b0c64eea-e874-4ae2-a6db-61f394284ee3","Type":"ContainerStarted","Data":"371c057669f661b724844a98c5db3768f9e42978c92613ad785c63235f515b18"} Oct 30 07:12:42.904699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:42.904577 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:43.534676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:43.534624 5894 generic.go:356] "Generic (PLEG): container finished" podID="b0c64eea-e874-4ae2-a6db-61f394284ee3" containerID="371c057669f661b724844a98c5db3768f9e42978c92613ad785c63235f515b18" exitCode=0 Oct 30 07:12:43.534676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:43.534666 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" event={"ID":"b0c64eea-e874-4ae2-a6db-61f394284ee3","Type":"ContainerDied","Data":"371c057669f661b724844a98c5db3768f9e42978c92613ad785c63235f515b18"} Oct 30 07:12:44.028991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.028927 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:44.517756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.517676 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:44.697827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.697803 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:44.780925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.780899 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\" (UniqueName: \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\") pod \"b0c64eea-e874-4ae2-a6db-61f394284ee3\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " Oct 30 07:12:44.781143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.781127 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\") pod \"b0c64eea-e874-4ae2-a6db-61f394284ee3\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " Oct 30 07:12:44.781219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.781178 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\") pod \"b0c64eea-e874-4ae2-a6db-61f394284ee3\" (UID: \"b0c64eea-e874-4ae2-a6db-61f394284ee3\") " Oct 30 07:12:44.781609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.781576 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca" pod "b0c64eea-e874-4ae2-a6db-61f394284ee3" (UID: "b0c64eea-e874-4ae2-a6db-61f394284ee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:44.783262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.783227 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca" pod "b0c64eea-e874-4ae2-a6db-61f394284ee3" (UID: "b0c64eea-e874-4ae2-a6db-61f394284ee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:44.783650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.783617 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2" pod "b0c64eea-e874-4ae2-a6db-61f394284ee3" (UID: "b0c64eea-e874-4ae2-a6db-61f394284ee3"). InnerVolumeSpecName "kube-api-access-5bbc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:44.882931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.882896 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:44.882931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.882927 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbc2\" (UniqueName: \"kubernetes.io/projected/b0c64eea-e874-4ae2-a6db-61f394284ee3-kube-api-access-5bbc2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:44.882931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:44.882937 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c64eea-e874-4ae2-a6db-61f394284ee3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:45.548542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:45.548512 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" Oct 30 07:12:45.549067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:45.548547 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p" event={"ID":"b0c64eea-e874-4ae2-a6db-61f394284ee3","Type":"ContainerDied","Data":"101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9"} Oct 30 07:12:45.549067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:45.548595 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101ab74aff4fd2e5b4ab02331968bbdf66886f61c14377028e353951f3e7f1a9" Oct 30 07:12:46.117487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.117451 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:46.583082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.583040 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx"] Oct 30 07:12:46.583854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.583833 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b0c64eea-e874-4ae2-a6db-61f394284ee3" containerName="revoke-keys" Oct 30 07:12:46.583935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.583856 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c64eea-e874-4ae2-a6db-61f394284ee3" containerName="revoke-keys" Oct 30 07:12:46.584065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.584043 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c64eea-e874-4ae2-a6db-61f394284ee3" containerName="revoke-keys" Oct 30 07:12:46.587739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.587709 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.608761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.608729 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx"] Oct 30 07:12:46.618942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.618908 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:46.618942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.618920 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:46.619122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.618953 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:46.620799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.620777 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:46.702463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.702430 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8kh\" (UniqueName: \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.702734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.702483 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.702734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.702517 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.804075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.804041 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8kh\" (UniqueName: \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.804225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.804172 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.804268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.804245 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.804982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.804955 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.806492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.806474 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.829976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.829942 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8kh\" (UniqueName: \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\") pod \"maas-api-revoke-sub-86cd2e7b-8m4bx\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.901822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.901790 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:46.944197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:46.944169 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:12:47.006587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006511 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006573 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006685 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006713 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006748 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006807 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006853 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006884 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.006989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.006926 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\") pod \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\" (UID: \"12eae9a0-97ce-438f-8dfd-45b6a83319a8\") " Oct 30 07:12:47.007543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.007523 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:47.007770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.007754 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:47.009888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.008248 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:47.009888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.008447 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:47.009888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.009013 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:47.011279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.011254 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "kube-api-access-96kbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:47.011369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.011314 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:47.012463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.012380 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:12:47.012720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.012692 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo" pod "12eae9a0-97ce-438f-8dfd-45b6a83319a8" (UID: "12eae9a0-97ce-438f-8dfd-45b6a83319a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109102 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109138 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109152 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109166 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109179 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109191 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109203 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96kbv\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-kube-api-access-96kbv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109217 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.110874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.109228 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12eae9a0-97ce-438f-8dfd-45b6a83319a8-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:47.210044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.210010 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx"] Oct 30 07:12:47.210372 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:47.210349 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda85f1f9d_cc0e_4639_9cd8_3d1f63355cd7.slice/crio-b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22 WatchSource:0}: Error finding container b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22: Status 404 returned error can't find the container with id b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22 Oct 30 07:12:47.563736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.563702 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" event={"ID":"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7","Type":"ContainerStarted","Data":"b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22"} Oct 30 07:12:47.565474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.565446 5894 generic.go:356] "Generic (PLEG): container finished" podID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerID="78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39" exitCode=0 Oct 30 07:12:47.565602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.565489 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" event={"ID":"12eae9a0-97ce-438f-8dfd-45b6a83319a8","Type":"ContainerDied","Data":"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39"} Oct 30 07:12:47.565602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.565510 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" Oct 30 07:12:47.565602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.565535 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n" event={"ID":"12eae9a0-97ce-438f-8dfd-45b6a83319a8","Type":"ContainerDied","Data":"e665afb173e8e764e98639c46738eb852447575756d045d9e87917346ce5240e"} Oct 30 07:12:47.565602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.565558 5894 scope.go:122] "RemoveContainer" containerID="78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39" Oct 30 07:12:47.576212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.576187 5894 scope.go:122] "RemoveContainer" containerID="78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39" Oct 30 07:12:47.576542 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:12:47.576519 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39\": container with ID starting with 78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39 not found: ID does not exist" containerID="78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39" Oct 30 07:12:47.576629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.576552 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39"} err="failed to get container status \"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39\": rpc error: code = NotFound desc = could not find container \"78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39\": container with ID starting with 78cda5a64c915bd5a77459348cb76f26e078fdd275215b4a9c57b1275e56aa39 not found: ID does not exist" Oct 30 07:12:47.628221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.628188 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:12:47.642063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.641989 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-fjz4n"] Oct 30 07:12:47.735345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:47.735307 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:48.430335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.430259 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" path="/var/lib/kubelet/pods/12eae9a0-97ce-438f-8dfd-45b6a83319a8/volumes" Oct 30 07:12:48.574306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.574269 5894 generic.go:356] "Generic (PLEG): container finished" podID="a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" containerID="9351320717c596a9ce1ee772f4a3e4d1df26da6d132b3f43683a509db75ea5be" exitCode=0 Oct 30 07:12:48.574453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.574309 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" event={"ID":"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7","Type":"ContainerDied","Data":"9351320717c596a9ce1ee772f4a3e4d1df26da6d132b3f43683a509db75ea5be"} Oct 30 07:12:48.641602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.641572 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4"] Oct 30 07:12:48.642185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.642166 5894 cpu_manager.go:414] "containerMap: removing container" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" Oct 30 07:12:48.642247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.642187 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" Oct 30 07:12:48.642324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.642314 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="12eae9a0-97ce-438f-8dfd-45b6a83319a8" containerName="istio-proxy" Oct 30 07:12:48.645934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.645916 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.664853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.664824 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4"] Oct 30 07:12:48.726474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.726382 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvbq\" (UniqueName: \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.726474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.726449 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.726718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.726595 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.827645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.827611 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.827826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.827694 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvbq\" (UniqueName: \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.827826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.827744 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.828400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.828371 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.830003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.829979 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.868236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.868201 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvbq\" (UniqueName: \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\") pod \"maas-api-revoke-sub-7c80d241-c68s4\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:48.959140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:48.959098 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:49.261693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.261656 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4"] Oct 30 07:12:49.261940 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:49.261918 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984c2125_5e72_41b1_808f_bdd0ad4e6979.slice/crio-c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c WatchSource:0}: Error finding container c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c: Status 404 returned error can't find the container with id c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c Oct 30 07:12:49.408102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.408067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:49.582490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.582458 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" event={"ID":"984c2125-5e72-41b1-808f-bdd0ad4e6979","Type":"ContainerStarted","Data":"c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c"} Oct 30 07:12:49.723296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.723267 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:49.838957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.838895 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\") pod \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " Oct 30 07:12:49.838957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.838943 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\") pod \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " Oct 30 07:12:49.839114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.839098 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\" (UniqueName: \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\") pod \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\" (UID: \"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7\") " Oct 30 07:12:49.839409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.839373 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca" pod "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" (UID: "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:49.839656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.839615 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:49.841302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.841278 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh" pod "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" (UID: "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7"). InnerVolumeSpecName "kube-api-access-5s8kh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:49.841397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.841374 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca" pod "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" (UID: "a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:49.941115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.941082 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8kh\" (UniqueName: \"kubernetes.io/projected/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-kube-api-access-5s8kh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:49.941115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:49.941112 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:50.594034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:50.593782 5894 generic.go:356] "Generic (PLEG): container finished" podID="984c2125-5e72-41b1-808f-bdd0ad4e6979" containerID="000ae1e872b7a3d84aeef9005d7707d127235ff238663be84096ee8ee56e2156" exitCode=0 Oct 30 07:12:50.594034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:50.593854 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" event={"ID":"984c2125-5e72-41b1-808f-bdd0ad4e6979","Type":"ContainerDied","Data":"000ae1e872b7a3d84aeef9005d7707d127235ff238663be84096ee8ee56e2156"} Oct 30 07:12:50.597097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:50.596230 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" event={"ID":"a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7","Type":"ContainerDied","Data":"b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22"} Oct 30 07:12:50.597097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:50.596267 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx" Oct 30 07:12:50.597097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:50.596286 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4cc49312e3fa2b0ba4198b9da6f2800fff2cebd8a34ed670fe139636b4c9e22" Oct 30 07:12:51.163184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:51.163155 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:52.361365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.361335 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:52.518272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.518020 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\") pod \"984c2125-5e72-41b1-808f-bdd0ad4e6979\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " Oct 30 07:12:52.518272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.518093 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\" (UniqueName: \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\") pod \"984c2125-5e72-41b1-808f-bdd0ad4e6979\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " Oct 30 07:12:52.518272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.518142 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\") pod \"984c2125-5e72-41b1-808f-bdd0ad4e6979\" (UID: \"984c2125-5e72-41b1-808f-bdd0ad4e6979\") " Oct 30 07:12:52.518590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.518557 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca" pod "984c2125-5e72-41b1-808f-bdd0ad4e6979" (UID: "984c2125-5e72-41b1-808f-bdd0ad4e6979"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:52.518764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.518741 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:52.522418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.522056 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq" pod "984c2125-5e72-41b1-808f-bdd0ad4e6979" (UID: "984c2125-5e72-41b1-808f-bdd0ad4e6979"). InnerVolumeSpecName "kube-api-access-msvbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:52.526342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.525796 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca" pod "984c2125-5e72-41b1-808f-bdd0ad4e6979" (UID: "984c2125-5e72-41b1-808f-bdd0ad4e6979"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:52.620086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.620043 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvbq\" (UniqueName: \"kubernetes.io/projected/984c2125-5e72-41b1-808f-bdd0ad4e6979-kube-api-access-msvbq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:52.620086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.620072 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984c2125-5e72-41b1-808f-bdd0ad4e6979-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:52.627086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.627058 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" Oct 30 07:12:52.627343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.627081 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4" event={"ID":"984c2125-5e72-41b1-808f-bdd0ad4e6979","Type":"ContainerDied","Data":"c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c"} Oct 30 07:12:52.627447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.627372 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4cc26205a9db5a62264d02a5641f8e4c7319225215ff698c96f09e1c7249c6c" Oct 30 07:12:52.834244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:52.834211 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:53.674960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.674922 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb"] Oct 30 07:12:53.675555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675538 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" containerName="revoke-keys" Oct 30 07:12:53.675555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675555 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" containerName="revoke-keys" Oct 30 07:12:53.675660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675601 5894 cpu_manager.go:414] "containerMap: removing container" podUID="984c2125-5e72-41b1-808f-bdd0ad4e6979" containerName="revoke-keys" Oct 30 07:12:53.675660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675606 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984c2125-5e72-41b1-808f-bdd0ad4e6979" containerName="revoke-keys" Oct 30 07:12:53.675730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675710 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="984c2125-5e72-41b1-808f-bdd0ad4e6979" containerName="revoke-keys" Oct 30 07:12:53.675730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.675724 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" containerName="revoke-keys" Oct 30 07:12:53.679586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.679565 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.696991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.696954 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:53.696991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.696961 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:53.697966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.697941 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb"] Oct 30 07:12:53.708751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.708720 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:53.709274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.709251 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:53.733490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.733459 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpc57\" (UniqueName: \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.733658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.733620 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.733702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.733686 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.834769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.834735 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.834926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.834780 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.834926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.834868 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpc57\" (UniqueName: \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.835499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.835471 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.837285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.837250 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.861833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.861796 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpc57\" (UniqueName: \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\") pod \"maas-api-revoke-sub-c9d8fddf-2vbtb\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:53.999033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:53.998949 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:54.309972 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:54.309935 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7de361_83df_452e_8725_840cb63d2824.slice/crio-448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0 WatchSource:0}: Error finding container 448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0: Status 404 returned error can't find the container with id 448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0 Oct 30 07:12:54.311349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:54.311322 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb"] Oct 30 07:12:54.347831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:54.347801 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:54.649834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:54.649790 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" event={"ID":"1d7de361-83df-452e-8725-840cb63d2824","Type":"ContainerStarted","Data":"d29cbce993a2c18ad76072b16f999770d21197858b5e2bc9d1b0081f73f61fdc"} Oct 30 07:12:54.650047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:54.649888 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" event={"ID":"1d7de361-83df-452e-8725-840cb63d2824","Type":"ContainerStarted","Data":"448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0"} Oct 30 07:12:54.696381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:54.696326 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" podStartSLOduration=1.696311312 podStartE2EDuration="1.696311312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:12:54.694153124 +0000 UTC m=+938.385070560" watchObservedRunningTime="2026-10-30 07:12:54.696311312 +0000 UTC m=+938.387228749" Oct 30 07:12:55.658840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:55.658805 5894 generic.go:356] "Generic (PLEG): container finished" podID="1d7de361-83df-452e-8725-840cb63d2824" containerID="d29cbce993a2c18ad76072b16f999770d21197858b5e2bc9d1b0081f73f61fdc" exitCode=0 Oct 30 07:12:55.659018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:55.658889 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" event={"ID":"1d7de361-83df-452e-8725-840cb63d2824","Type":"ContainerDied","Data":"d29cbce993a2c18ad76072b16f999770d21197858b5e2bc9d1b0081f73f61fdc"} Oct 30 07:12:55.934836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:55.934754 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:56.817396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.817365 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:56.869746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.869718 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\") pod \"1d7de361-83df-452e-8725-840cb63d2824\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " Oct 30 07:12:56.869913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.869789 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\") pod \"1d7de361-83df-452e-8725-840cb63d2824\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " Oct 30 07:12:56.869913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.869820 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\" (UniqueName: \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\") pod \"1d7de361-83df-452e-8725-840cb63d2824\" (UID: \"1d7de361-83df-452e-8725-840cb63d2824\") " Oct 30 07:12:56.870144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.870122 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca" pod "1d7de361-83df-452e-8725-840cb63d2824" (UID: "1d7de361-83df-452e-8725-840cb63d2824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:12:56.870310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.870290 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d7de361-83df-452e-8725-840cb63d2824-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:56.871971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.871942 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57" pod "1d7de361-83df-452e-8725-840cb63d2824" (UID: "1d7de361-83df-452e-8725-840cb63d2824"). InnerVolumeSpecName "kube-api-access-rpc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:12:56.871971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.871972 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca" pod "1d7de361-83df-452e-8725-840cb63d2824" (UID: "1d7de361-83df-452e-8725-840cb63d2824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:12:56.971166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.971080 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d7de361-83df-452e-8725-840cb63d2824-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:56.971166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:56.971111 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpc57\" (UniqueName: \"kubernetes.io/projected/1d7de361-83df-452e-8725-840cb63d2824-kube-api-access-rpc57\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:12:57.061232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:57.061199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:57.620274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:57.620246 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:57.674320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:57.674286 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" Oct 30 07:12:57.674320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:57.674297 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb" event={"ID":"1d7de361-83df-452e-8725-840cb63d2824","Type":"ContainerDied","Data":"448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0"} Oct 30 07:12:57.674545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:57.674354 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="448da68677955922fc2d3566bb6293dbc03744c6966952595899b57cade430b0" Oct 30 07:12:59.080495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.080454 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:12:59.081034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.081019 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1d7de361-83df-452e-8725-840cb63d2824" containerName="revoke-keys" Oct 30 07:12:59.081080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.081037 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d7de361-83df-452e-8725-840cb63d2824" containerName="revoke-keys" Oct 30 07:12:59.081156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.081147 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d7de361-83df-452e-8725-840cb63d2824" containerName="revoke-keys" Oct 30 07:12:59.084614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.084597 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.122497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.122461 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:12:59.194715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.194663 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.194715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.194713 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.194928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.194820 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.194928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.194871 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.195041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.194920 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.195041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.195020 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.195103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.195052 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflsz\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.195103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.195083 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.195165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.195126 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.269226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.269191 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:12:59.296320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296286 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296326 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296455 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296507 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296533 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296580 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296662 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflsz\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296673 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296723 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296778 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.296904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296881 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.297008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.296990 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.297052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.297026 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.297173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.297148 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.299197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.299172 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.299311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.299239 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.323015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.322971 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflsz\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.323449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.323423 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\") pod \"e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.401165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.401082 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:12:59.404319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.404290 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:12:59.408704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.408672 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.438717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.438679 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:12:59.499817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.499785 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rqc\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.499817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.499824 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.499956 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500005 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500055 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500100 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500146 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500284 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.500372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.500324 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601180 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601228 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601289 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rqc\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601322 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601402 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601443 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601467 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601539 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601741 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.601890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.601843 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.602113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.602072 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.602237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.602137 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.602237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.602150 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.602357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.602273 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.604022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.603998 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.604242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.604221 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.626644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.626595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.626644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.626625 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rqc\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.678265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.678195 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5"] Oct 30 07:12:59.683458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.683435 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.700875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.700823 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:12:59.700875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.700861 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:12:59.701098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.700923 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:12:59.701098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.700948 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:12:59.701392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.701368 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5"] Oct 30 07:12:59.751696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.751663 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:12:59.755463 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:12:59.755432 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda894a0ef_dc4a_438a_95cd_3cfa4cebadd0.slice/crio-ac5fd5bbded235ee7a6f32867472c222d143313563673c00e38fd75bf3427149 WatchSource:0}: Error finding container ac5fd5bbded235ee7a6f32867472c222d143313563673c00e38fd75bf3427149: Status 404 returned error can't find the container with id ac5fd5bbded235ee7a6f32867472c222d143313563673c00e38fd75bf3427149 Oct 30 07:12:59.755770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.755743 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:12:59.761321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.761285 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:59.761426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.761353 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:59.761426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.761381 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:12:59.804166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.804130 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.804381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.804356 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.804443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.804415 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vncv\" (UniqueName: \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.905867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.905832 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.906067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.905977 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.906067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.906011 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vncv\" (UniqueName: \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.906513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.906487 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.908413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.908391 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.931342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.931262 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vncv\" (UniqueName: \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\") pod \"maas-api-revoke-sub-c514af58-khbk5\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:12:59.998275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:12:59.998229 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:13:00.084993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.084796 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:13:00.087312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.087274 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.087442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.087361 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.087442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.087401 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:00.328704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.328030 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5"] Oct 30 07:13:00.727053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.727015 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" event={"ID":"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0","Type":"ContainerStarted","Data":"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34"} Oct 30 07:13:00.727311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.727295 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" event={"ID":"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0","Type":"ContainerStarted","Data":"ac5fd5bbded235ee7a6f32867472c222d143313563673c00e38fd75bf3427149"} Oct 30 07:13:00.728809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.728780 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" event={"ID":"d64b6301-19b8-4009-8854-77f1e00b7d2a","Type":"ContainerStarted","Data":"c1d9600e58f283e7a5d04dbdc5a47f91b4eccd15ef352804f41de4139cf4dc8b"} Oct 30 07:13:00.730793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.730761 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" event={"ID":"4b7708c6-f652-4435-8ed2-4e36488ffd68","Type":"ContainerStarted","Data":"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d"} Oct 30 07:13:00.730894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.730822 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" event={"ID":"4b7708c6-f652-4435-8ed2-4e36488ffd68","Type":"ContainerStarted","Data":"240158716c5ec1c68830d704ae7314f514e60e7fb08495e71fbdaa7b02f0ac45"} Oct 30 07:13:00.753446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.752777 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:13:00.782659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.780172 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podStartSLOduration=1.7801517850000002 podStartE2EDuration="1.780151785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.77823885 +0000 UTC m=+944.469156287" watchObservedRunningTime="2026-10-30 07:13:00.780151785 +0000 UTC m=+944.471069223" Oct 30 07:13:00.837710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:00.837656 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podStartSLOduration=1.8376254539999999 podStartE2EDuration="1.837625454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:00.833656916 +0000 UTC m=+944.524574354" watchObservedRunningTime="2026-10-30 07:13:00.837625454 +0000 UTC m=+944.528542891" Oct 30 07:13:01.211660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.211605 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:01.414284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.413005 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:13:01.423945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.423896 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.424268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.424211 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:01.740073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.740038 5894 generic.go:356] "Generic (PLEG): container finished" podID="d64b6301-19b8-4009-8854-77f1e00b7d2a" containerID="f442045541c0c27d8cf5a564b0a35c27c4dba514b916a64dfe3aafefc9a9038d" exitCode=0 Oct 30 07:13:01.740247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.740124 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" event={"ID":"d64b6301-19b8-4009-8854-77f1e00b7d2a","Type":"ContainerDied","Data":"f442045541c0c27d8cf5a564b0a35c27c4dba514b916a64dfe3aafefc9a9038d"} Oct 30 07:13:01.753332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.753258 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:01.753332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:01.753310 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:02.402085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:02.402040 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.402700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:02.402667 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:02.752509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:02.752476 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:02.752674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:02.752557 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:02.959586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:02.959461 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:03.049997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.049968 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:13:03.062509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.062478 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\") pod \"d64b6301-19b8-4009-8854-77f1e00b7d2a\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " Oct 30 07:13:03.062720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.062675 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\" (UniqueName: \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\") pod \"d64b6301-19b8-4009-8854-77f1e00b7d2a\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " Oct 30 07:13:03.062795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.062740 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\") pod \"d64b6301-19b8-4009-8854-77f1e00b7d2a\" (UID: \"d64b6301-19b8-4009-8854-77f1e00b7d2a\") " Oct 30 07:13:03.063176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.063153 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca" pod "d64b6301-19b8-4009-8854-77f1e00b7d2a" (UID: "d64b6301-19b8-4009-8854-77f1e00b7d2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:03.065435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.065392 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca" pod "d64b6301-19b8-4009-8854-77f1e00b7d2a" (UID: "d64b6301-19b8-4009-8854-77f1e00b7d2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:03.065435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.065409 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv" pod "d64b6301-19b8-4009-8854-77f1e00b7d2a" (UID: "d64b6301-19b8-4009-8854-77f1e00b7d2a"). InnerVolumeSpecName "kube-api-access-2vncv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:03.134776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.134732 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn"] Oct 30 07:13:03.135897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.135869 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d64b6301-19b8-4009-8854-77f1e00b7d2a" containerName="revoke-keys" Oct 30 07:13:03.136156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.136130 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d64b6301-19b8-4009-8854-77f1e00b7d2a" containerName="revoke-keys" Oct 30 07:13:03.136466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.136452 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d64b6301-19b8-4009-8854-77f1e00b7d2a" containerName="revoke-keys" Oct 30 07:13:03.142807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.140887 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.143732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.143704 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn"] Oct 30 07:13:03.164279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164251 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvp2q\" (UniqueName: \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.164410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164385 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.164484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164462 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.164538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164528 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:03.164578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164540 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vncv\" (UniqueName: \"kubernetes.io/projected/d64b6301-19b8-4009-8854-77f1e00b7d2a-kube-api-access-2vncv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:03.164578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.164552 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d64b6301-19b8-4009-8854-77f1e00b7d2a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:03.265849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.265805 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvp2q\" (UniqueName: \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.266044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.265935 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.266044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.265979 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.266738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.266712 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.270459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.270420 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.330723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.328540 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvp2q\" (UniqueName: \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\") pod \"maas-api-revoke-sub-51c9f5aa-6gwdn\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.405071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.405027 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.405709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.405674 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:03.461180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.461136 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:03.752839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.752749 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:03.752839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.752804 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:03.759681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.756846 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" event={"ID":"d64b6301-19b8-4009-8854-77f1e00b7d2a","Type":"ContainerDied","Data":"c1d9600e58f283e7a5d04dbdc5a47f91b4eccd15ef352804f41de4139cf4dc8b"} Oct 30 07:13:03.759681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.756905 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1d9600e58f283e7a5d04dbdc5a47f91b4eccd15ef352804f41de4139cf4dc8b" Oct 30 07:13:03.759681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.756969 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5" Oct 30 07:13:03.825756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:03.825725 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn"] Oct 30 07:13:03.835097 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:03.835055 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5236f9_91d4_4017_9834_eeb242fa7799.slice/crio-761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3 WatchSource:0}: Error finding container 761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3: Status 404 returned error can't find the container with id 761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3 Oct 30 07:13:04.427720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.418219 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.427720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.418317 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:04.731763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.731678 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:04.753669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.752985 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:04.753669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.753052 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:04.765203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.765167 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" event={"ID":"ee5236f9-91d4-4017-9834-eeb242fa7799","Type":"ContainerStarted","Data":"3c4aee608abce838a2d434ff5a3d5db8ffe68a78cc454442585e53dab0496817"} Oct 30 07:13:04.765373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.765227 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" event={"ID":"ee5236f9-91d4-4017-9834-eeb242fa7799","Type":"ContainerStarted","Data":"761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3"} Oct 30 07:13:04.847339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:04.847008 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" podStartSLOduration=1.846980032 podStartE2EDuration="1.846980032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:04.82702938 +0000 UTC m=+948.517946817" watchObservedRunningTime="2026-10-30 07:13:04.846980032 +0000 UTC m=+948.537897474" Oct 30 07:13:04.942661 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:13:04.939592 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:13:05.402128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.402084 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.402455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.402424 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:05.752103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.752013 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:05.752103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.752083 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:05.804107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.804067 5894 generic.go:356] "Generic (PLEG): container finished" podID="ee5236f9-91d4-4017-9834-eeb242fa7799" containerID="3c4aee608abce838a2d434ff5a3d5db8ffe68a78cc454442585e53dab0496817" exitCode=0 Oct 30 07:13:05.804421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.804402 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" event={"ID":"ee5236f9-91d4-4017-9834-eeb242fa7799","Type":"ContainerDied","Data":"3c4aee608abce838a2d434ff5a3d5db8ffe68a78cc454442585e53dab0496817"} Oct 30 07:13:05.858245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.858210 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 07:13:05.863171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.863148 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.899423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.899392 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"distinct-w3-2f845b-kserve-self-signed-certs\"" Oct 30 07:13:05.926296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.926267 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 07:13:05.934981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.934941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:05.954045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.953994 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.954045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.954046 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.954314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.954198 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk5r5\" (UniqueName: \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.954314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.954245 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.954314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.954269 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:05.954473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:05.954375 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055339 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk5r5\" (UniqueName: \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055405 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055463 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055495 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055510 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.055930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.055909 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.056287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.056251 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.056566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.056529 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.058907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.058616 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.059203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.059161 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.092654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.085694 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk5r5\" (UniqueName: \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\") pod \"distinct-w3-2f845b-kserve-79f848d597-ffrqh\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.214680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.213866 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:06.402196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.402153 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.402379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.402226 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 07:13:06.567086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.567056 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 07:13:06.567422 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:06.567399 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe76c9f_e2ef_471b_a29b_5d9caee9947a.slice/crio-bd6c38cf04d4b680cf2797518fa54e274d244fb08cec4a19ded462f5931f06bc WatchSource:0}: Error finding container bd6c38cf04d4b680cf2797518fa54e274d244fb08cec4a19ded462f5931f06bc: Status 404 returned error can't find the container with id bd6c38cf04d4b680cf2797518fa54e274d244fb08cec4a19ded462f5931f06bc Oct 30 07:13:06.642114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.642063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:06.752797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.752759 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 07:13:06.753215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.752828 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 07:13:06.815302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.815195 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" event={"ID":"fbe76c9f-e2ef-471b-a29b-5d9caee9947a","Type":"ContainerStarted","Data":"c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef"} Oct 30 07:13:06.815302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.815249 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" event={"ID":"fbe76c9f-e2ef-471b-a29b-5d9caee9947a","Type":"ContainerStarted","Data":"bd6c38cf04d4b680cf2797518fa54e274d244fb08cec4a19ded462f5931f06bc"} Oct 30 07:13:06.867439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.867331 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" podStartSLOduration=1.867310408 podStartE2EDuration="1.867310408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:06.862110527 +0000 UTC m=+950.553027969" watchObservedRunningTime="2026-10-30 07:13:06.867310408 +0000 UTC m=+950.558227849" Oct 30 07:13:06.984283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:06.984254 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:07.075242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.074097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\" (UniqueName: \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\") pod \"ee5236f9-91d4-4017-9834-eeb242fa7799\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " Oct 30 07:13:07.075242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.074137 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\") pod \"ee5236f9-91d4-4017-9834-eeb242fa7799\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " Oct 30 07:13:07.075242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.074175 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\") pod \"ee5236f9-91d4-4017-9834-eeb242fa7799\" (UID: \"ee5236f9-91d4-4017-9834-eeb242fa7799\") " Oct 30 07:13:07.075844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.075615 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca" pod "ee5236f9-91d4-4017-9834-eeb242fa7799" (UID: "ee5236f9-91d4-4017-9834-eeb242fa7799"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:07.078428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.078399 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca" pod "ee5236f9-91d4-4017-9834-eeb242fa7799" (UID: "ee5236f9-91d4-4017-9834-eeb242fa7799"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:07.080826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.080801 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q" pod "ee5236f9-91d4-4017-9834-eeb242fa7799" (UID: "ee5236f9-91d4-4017-9834-eeb242fa7799"). InnerVolumeSpecName "kube-api-access-xvp2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:07.176500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.176458 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvp2q\" (UniqueName: \"kubernetes.io/projected/ee5236f9-91d4-4017-9834-eeb242fa7799-kube-api-access-xvp2q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:07.176500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.176501 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:07.176724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.176519 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5236f9-91d4-4017-9834-eeb242fa7799-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:07.214552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.214513 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:07.314098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.314060 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:13:07.314947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.314917 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ee5236f9-91d4-4017-9834-eeb242fa7799" containerName="revoke-keys" Oct 30 07:13:07.314947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.314945 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5236f9-91d4-4017-9834-eeb242fa7799" containerName="revoke-keys" Oct 30 07:13:07.315155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.315146 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5236f9-91d4-4017-9834-eeb242fa7799" containerName="revoke-keys" Oct 30 07:13:07.320058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.319477 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.353710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.352093 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:13:07.382385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382328 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382392 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382427 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382550 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382583 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrmz\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382647 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382724 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.382816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382782 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.383039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.382859 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484174 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484248 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484284 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484315 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484429 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484459 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrmz\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484499 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484577 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.484733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.484650 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.485406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.485379 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.486608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.485779 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.486758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.486142 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.486758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.486487 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.487093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.487072 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.488202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.488076 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.489005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.488964 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.512917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.512848 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrmz\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.513293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.513235 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.562927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.562896 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:13:07.563607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.563342 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:13:07.564576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.564518 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:13:07.691993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.691908 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:07.767745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.767709 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755"] Oct 30 07:13:07.780328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.776456 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:07.787730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.783519 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:13:07.787730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.784654 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:13:07.806388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.805193 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755"] Oct 30 07:13:07.815705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.815004 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:13:07.836269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.836239 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" Oct 30 07:13:07.836269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.836256 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn" event={"ID":"ee5236f9-91d4-4017-9834-eeb242fa7799","Type":"ContainerDied","Data":"761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3"} Oct 30 07:13:07.836497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.836315 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="761abb55e02737b87f2b3fefe535fa3bc014a2a9d78b7664d6cb4cc9d769cea3" Oct 30 07:13:07.905955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.905917 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8gg\" (UniqueName: \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:07.906157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.906054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:07.906367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:07.906345 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.007791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.007689 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.007791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.007785 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8gg\" (UniqueName: \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.008026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.007871 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.009018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.008988 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.010889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.010850 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.034595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.034565 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8gg\" (UniqueName: \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\") pod \"maas-api-revoke-sub-8387e190-7p755\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.044338 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:08.044310 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d1cfb5_9803_4356_995b_815e0dbdef5c.slice/crio-85c4184e9ea1730117b19d21b776086afe1cfada9317283b9b6ee81cef50f609 WatchSource:0}: Error finding container 85c4184e9ea1730117b19d21b776086afe1cfada9317283b9b6ee81cef50f609: Status 404 returned error can't find the container with id 85c4184e9ea1730117b19d21b776086afe1cfada9317283b9b6ee81cef50f609 Oct 30 07:13:08.044725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.044701 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:13:08.046784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.046755 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.046849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.046819 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.046903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.046847 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:08.122700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.122671 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:08.280691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.280555 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:08.439436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.439405 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755"] Oct 30 07:13:08.439865 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:08.439840 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf304254c_3904_4e9b_b52c_f49f1505c9f4.slice/crio-84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311 WatchSource:0}: Error finding container 84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311: Status 404 returned error can't find the container with id 84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311 Oct 30 07:13:08.844007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.843970 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" event={"ID":"f304254c-3904-4e9b-b52c-f49f1505c9f4","Type":"ContainerStarted","Data":"84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311"} Oct 30 07:13:08.845909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.845871 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" event={"ID":"83d1cfb5-9803-4356-995b-815e0dbdef5c","Type":"ContainerStarted","Data":"3c4b2c2d0e2554df8a693ab5147e63c92f222a2c95c20822fc82988d119094cc"} Oct 30 07:13:08.846081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.845926 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" event={"ID":"83d1cfb5-9803-4356-995b-815e0dbdef5c","Type":"ContainerStarted","Data":"85c4184e9ea1730117b19d21b776086afe1cfada9317283b9b6ee81cef50f609"} Oct 30 07:13:08.918422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:08.918379 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podStartSLOduration=1.9183658019999998 podStartE2EDuration="1.918365802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:08.91805895 +0000 UTC m=+952.608976387" watchObservedRunningTime="2026-10-30 07:13:08.918365802 +0000 UTC m=+952.609283294" Oct 30 07:13:09.695783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:09.695693 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:09.696412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:09.696081 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:09.696412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:09.696147 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:09.855209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:09.855177 5894 generic.go:356] "Generic (PLEG): container finished" podID="f304254c-3904-4e9b-b52c-f49f1505c9f4" containerID="0225b732fdb54b49936e117c5c781209c6068cec24e5de7f25f6b80975278f97" exitCode=0 Oct 30 07:13:09.855673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:09.855265 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" event={"ID":"f304254c-3904-4e9b-b52c-f49f1505c9f4","Type":"ContainerDied","Data":"0225b732fdb54b49936e117c5c781209c6068cec24e5de7f25f6b80975278f97"} Oct 30 07:13:10.038381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.038345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:10.094091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.088179 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr"] Oct 30 07:13:10.102669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.100377 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.129892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.129805 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr"] Oct 30 07:13:10.136980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.136935 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.137151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.137132 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pksf4\" (UniqueName: \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.137229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.137213 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.238969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.238923 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pksf4\" (UniqueName: \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.239184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.239017 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.239184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.239093 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.239746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.239721 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.241468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.241444 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.264423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.264394 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pksf4\" (UniqueName: \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\") pod \"maas-api-revoke-sub-add5574c-7stnr\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.470910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.470810 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:10.696681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.696614 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:10.696846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.696704 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:10.801298 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:10.801261 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5930a586_bbc4_4e42_bbb9_606a4d276100.slice/crio-bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3 WatchSource:0}: Error finding container bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3: Status 404 returned error can't find the container with id bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3 Oct 30 07:13:10.801472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.801438 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr"] Oct 30 07:13:10.854812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.854777 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:13:10.863070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:10.863030 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" event={"ID":"5930a586-bbc4-4e42-bbb9-606a4d276100","Type":"ContainerStarted","Data":"bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3"} Oct 30 07:13:11.204621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.204597 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:11.252929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.252899 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\") pod \"f304254c-3904-4e9b-b52c-f49f1505c9f4\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " Oct 30 07:13:11.253100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.253010 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\" (UniqueName: \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\") pod \"f304254c-3904-4e9b-b52c-f49f1505c9f4\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " Oct 30 07:13:11.253100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.253059 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\") pod \"f304254c-3904-4e9b-b52c-f49f1505c9f4\" (UID: \"f304254c-3904-4e9b-b52c-f49f1505c9f4\") " Oct 30 07:13:11.253791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.253764 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca" pod "f304254c-3904-4e9b-b52c-f49f1505c9f4" (UID: "f304254c-3904-4e9b-b52c-f49f1505c9f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:11.255902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.255869 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca" pod "f304254c-3904-4e9b-b52c-f49f1505c9f4" (UID: "f304254c-3904-4e9b-b52c-f49f1505c9f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:11.255999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.255976 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg" pod "f304254c-3904-4e9b-b52c-f49f1505c9f4" (UID: "f304254c-3904-4e9b-b52c-f49f1505c9f4"). InnerVolumeSpecName "kube-api-access-lt8gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:11.355308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.355276 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:11.355308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.355304 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8gg\" (UniqueName: \"kubernetes.io/projected/f304254c-3904-4e9b-b52c-f49f1505c9f4-kube-api-access-lt8gg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:11.355519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.355314 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f304254c-3904-4e9b-b52c-f49f1505c9f4-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:11.656658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.656545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:11.697660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.697057 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:11.697660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.697119 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:11.898845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.893037 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" event={"ID":"5930a586-bbc4-4e42-bbb9-606a4d276100","Type":"ContainerStarted","Data":"ec84a1b2f8e71c8b42008373f91aba1b9228a747cb8a7d27ec5acedff5a67684"} Oct 30 07:13:11.898845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.896377 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" event={"ID":"f304254c-3904-4e9b-b52c-f49f1505c9f4","Type":"ContainerDied","Data":"84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311"} Oct 30 07:13:11.898845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.896433 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84542fad46ea28725e33503d455d928900c8c7a3c3d0fd693eb95df115952311" Oct 30 07:13:11.898845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.896509 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755" Oct 30 07:13:11.968288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:11.968231 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" podStartSLOduration=1.96821107 podStartE2EDuration="1.96821107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:11.961562584 +0000 UTC m=+955.652480027" watchObservedRunningTime="2026-10-30 07:13:11.96821107 +0000 UTC m=+955.659128513" Oct 30 07:13:12.693114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:12.693072 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:12.693444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:12.693406 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:12.922215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:12.922177 5894 generic.go:356] "Generic (PLEG): container finished" podID="5930a586-bbc4-4e42-bbb9-606a4d276100" containerID="ec84a1b2f8e71c8b42008373f91aba1b9228a747cb8a7d27ec5acedff5a67684" exitCode=0 Oct 30 07:13:12.922932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:12.922902 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" event={"ID":"5930a586-bbc4-4e42-bbb9-606a4d276100","Type":"ContainerDied","Data":"ec84a1b2f8e71c8b42008373f91aba1b9228a747cb8a7d27ec5acedff5a67684"} Oct 30 07:13:12.980851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:12.980747 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.010995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:13.010948 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.107659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:13.107443 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:13.697106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:13.697062 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:13.697392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:13.697361 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:14.154578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.154512 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 30 07:13:14.155807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.155782 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:13:14.195660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.175772 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 30 07:13:14.708123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.708083 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:14.708410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.708382 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:15.132268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.712352 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8"] Oct 30 07:13:15.141857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.141807 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f304254c-3904-4e9b-b52c-f49f1505c9f4" containerName="revoke-keys" Oct 30 07:13:15.150056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.150024 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f304254c-3904-4e9b-b52c-f49f1505c9f4" containerName="revoke-keys" Oct 30 07:13:15.151120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.151093 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f304254c-3904-4e9b-b52c-f49f1505c9f4" containerName="revoke-keys" Oct 30 07:13:15.153236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.153193 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.155509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.155451 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.227663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:14.967805 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:15.227663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.206516 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8"] Oct 30 07:13:15.227663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.206683 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.242655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.232223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:15.361069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.361035 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfr5z\" (UniqueName: \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.363851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.362804 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 07:13:15.364076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.364044 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.364278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.364257 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.430293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.430267 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:15.464965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.464930 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\" (UniqueName: \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\") pod \"5930a586-bbc4-4e42-bbb9-606a4d276100\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " Oct 30 07:13:15.465139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.465082 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\") pod \"5930a586-bbc4-4e42-bbb9-606a4d276100\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " Oct 30 07:13:15.465225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.465201 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\") pod \"5930a586-bbc4-4e42-bbb9-606a4d276100\" (UID: \"5930a586-bbc4-4e42-bbb9-606a4d276100\") " Oct 30 07:13:15.465431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.465408 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.466572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.465684 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca" pod "5930a586-bbc4-4e42-bbb9-606a4d276100" (UID: "5930a586-bbc4-4e42-bbb9-606a4d276100"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:15.466749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.466451 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.466845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.466676 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfr5z\" (UniqueName: \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.466903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.466845 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.467181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.467165 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:15.469165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.469125 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4" pod "5930a586-bbc4-4e42-bbb9-606a4d276100" (UID: "5930a586-bbc4-4e42-bbb9-606a4d276100"). InnerVolumeSpecName "kube-api-access-pksf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:15.469381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.469357 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca" pod "5930a586-bbc4-4e42-bbb9-606a4d276100" (UID: "5930a586-bbc4-4e42-bbb9-606a4d276100"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:15.469936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.469916 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.514095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.514070 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfr5z\" (UniqueName: \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\") pod \"maas-api-revoke-sub-146b6361-c7vt8\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:15.568732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.568696 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5930a586-bbc4-4e42-bbb9-606a4d276100-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:15.568732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.568725 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pksf4\" (UniqueName: \"kubernetes.io/projected/5930a586-bbc4-4e42-bbb9-606a4d276100-kube-api-access-pksf4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:15.693494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.693413 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 07:13:15.693494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.693476 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 07:13:15.727860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:15.727826 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:16.034805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.034772 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8"] Oct 30 07:13:16.034935 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:16.034844 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229d445f_11c6_4212_8e64_72b8da199703.slice/crio-6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c WatchSource:0}: Error finding container 6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c: Status 404 returned error can't find the container with id 6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c Oct 30 07:13:16.214450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.214419 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:16.222445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.222413 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" event={"ID":"229d445f-11c6-4212-8e64-72b8da199703","Type":"ContainerStarted","Data":"6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c"} Oct 30 07:13:16.224066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.224045 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" Oct 30 07:13:16.224225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.224041 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr" event={"ID":"5930a586-bbc4-4e42-bbb9-606a4d276100","Type":"ContainerDied","Data":"bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3"} Oct 30 07:13:16.224225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.224164 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdb32deb74f7845efc608bd3c0813f967c7273d2f5759fd7c057f72de744fbf3" Oct 30 07:13:16.229238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.229220 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:16.806151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:16.806117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:17.106759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.106729 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:17.107100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.107074 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:17.107955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.107939 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:13:17.226482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.226451 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 07:13:17.247987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.247956 5894 generic.go:356] "Generic (PLEG): container finished" podID="229d445f-11c6-4212-8e64-72b8da199703" containerID="f3b13df7c29b4d7adc648f00cb8380db0ad3dec8c545419251b9a62d6068eb32" exitCode=0 Oct 30 07:13:17.248401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:17.248375 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" event={"ID":"229d445f-11c6-4212-8e64-72b8da199703","Type":"ContainerDied","Data":"f3b13df7c29b4d7adc648f00cb8380db0ad3dec8c545419251b9a62d6068eb32"} Oct 30 07:13:18.091565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.091524 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl"] Oct 30 07:13:18.092448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.092423 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5930a586-bbc4-4e42-bbb9-606a4d276100" containerName="revoke-keys" Oct 30 07:13:18.092448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.092446 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5930a586-bbc4-4e42-bbb9-606a4d276100" containerName="revoke-keys" Oct 30 07:13:18.092680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.092559 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5930a586-bbc4-4e42-bbb9-606a4d276100" containerName="revoke-keys" Oct 30 07:13:18.096138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.096114 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.115294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.115262 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl"] Oct 30 07:13:18.202326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.202292 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgzd\" (UniqueName: \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.202528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.202459 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.202528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.202500 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.303597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.303556 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.304048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.303628 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.304048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.303810 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgzd\" (UniqueName: \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.305011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.304989 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.307332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.307308 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.332674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.332646 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgzd\" (UniqueName: \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\") pod \"maas-api-revoke-sub-60cb75cf-c96cl\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.413582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.413552 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:18.429870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.429844 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:18.505829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.505801 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\") pod \"229d445f-11c6-4212-8e64-72b8da199703\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " Oct 30 07:13:18.505966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.505907 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\" (UniqueName: \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\") pod \"229d445f-11c6-4212-8e64-72b8da199703\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " Oct 30 07:13:18.505966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.505960 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\") pod \"229d445f-11c6-4212-8e64-72b8da199703\" (UID: \"229d445f-11c6-4212-8e64-72b8da199703\") " Oct 30 07:13:18.506363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.506328 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca" pod "229d445f-11c6-4212-8e64-72b8da199703" (UID: "229d445f-11c6-4212-8e64-72b8da199703"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:18.507981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.507956 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca" pod "229d445f-11c6-4212-8e64-72b8da199703" (UID: "229d445f-11c6-4212-8e64-72b8da199703"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:18.508191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.508176 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z" pod "229d445f-11c6-4212-8e64-72b8da199703" (UID: "229d445f-11c6-4212-8e64-72b8da199703"). InnerVolumeSpecName "kube-api-access-jfr5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:18.539099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.539061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:18.607908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.607877 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/229d445f-11c6-4212-8e64-72b8da199703-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:18.607908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.607906 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfr5z\" (UniqueName: \"kubernetes.io/projected/229d445f-11c6-4212-8e64-72b8da199703-kube-api-access-jfr5z\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:18.608127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.607918 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/229d445f-11c6-4212-8e64-72b8da199703-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:18.722845 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:18.722812 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0665c44_138b_419b_89ec_2906d62bf6ee.slice/crio-fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423 WatchSource:0}: Error finding container fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423: Status 404 returned error can't find the container with id fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423 Oct 30 07:13:18.724147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:18.724122 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl"] Oct 30 07:13:19.264493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.264391 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" event={"ID":"e0665c44-138b-419b-89ec-2906d62bf6ee","Type":"ContainerStarted","Data":"6f40e6ad24ef90a6cab6448ab130fab86a69e90f30a029ac141c7e81e3e7d9a3"} Oct 30 07:13:19.264493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.264458 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" event={"ID":"e0665c44-138b-419b-89ec-2906d62bf6ee","Type":"ContainerStarted","Data":"fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423"} Oct 30 07:13:19.266105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.266077 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" event={"ID":"229d445f-11c6-4212-8e64-72b8da199703","Type":"ContainerDied","Data":"6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c"} Oct 30 07:13:19.266210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.266122 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6abb11e491077017b28d7cde30e7550c236401ae6108dc116248f69af550e43c" Oct 30 07:13:19.266210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.266083 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8" Oct 30 07:13:19.312075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.312011 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" podStartSLOduration=1.3119924219999999 podStartE2EDuration="1.311992422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:19.308710108 +0000 UTC m=+962.999627549" watchObservedRunningTime="2026-10-30 07:13:19.311992422 +0000 UTC m=+963.002909863" Oct 30 07:13:19.402508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.402473 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:13:19.403150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.403129 5894 cpu_manager.go:414] "containerMap: removing container" podUID="229d445f-11c6-4212-8e64-72b8da199703" containerName="revoke-keys" Oct 30 07:13:19.403276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.403151 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229d445f-11c6-4212-8e64-72b8da199703" containerName="revoke-keys" Oct 30 07:13:19.403361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.403345 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="229d445f-11c6-4212-8e64-72b8da199703" containerName="revoke-keys" Oct 30 07:13:19.406968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.406947 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.431336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.431301 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:13:19.518457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518380 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518416 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518439 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518463 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518583 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518607 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518685 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518721 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnmtk\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.518826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.518791 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620192 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620245 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnmtk\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620294 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620395 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620422 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620461 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620498 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620557 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.620602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620581 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.621029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.620997 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.621151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.621046 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.621151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.621081 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.621151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.621111 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.621323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.621234 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.622704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.622682 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.623028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.623010 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.645322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.645290 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.645590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.645561 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnmtk\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:19.722505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:19.722441 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:20.045360 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:20.045323 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c9448e_f07e_4cfc_af5a_a1edee7b4a55.slice/crio-eb43439b9974428d1bf0094bb00a043ca073936a66748f2b2d62bea6c70b6de5 WatchSource:0}: Error finding container eb43439b9974428d1bf0094bb00a043ca073936a66748f2b2d62bea6c70b6de5: Status 404 returned error can't find the container with id eb43439b9974428d1bf0094bb00a043ca073936a66748f2b2d62bea6c70b6de5 Oct 30 07:13:20.047332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.047300 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:13:20.047980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.047937 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.048085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.048032 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.048147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.048097 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:20.275471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.275435 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" event={"ID":"61c9448e-f07e-4cfc-af5a-a1edee7b4a55","Type":"ContainerStarted","Data":"0941b7b5657a87f88680d20d202d87406e5c0b4bc48bcb290768b22c2e1bcd31"} Oct 30 07:13:20.275718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.275527 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" event={"ID":"61c9448e-f07e-4cfc-af5a-a1edee7b4a55","Type":"ContainerStarted","Data":"eb43439b9974428d1bf0094bb00a043ca073936a66748f2b2d62bea6c70b6de5"} Oct 30 07:13:20.277478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.277443 5894 generic.go:356] "Generic (PLEG): container finished" podID="e0665c44-138b-419b-89ec-2906d62bf6ee" containerID="6f40e6ad24ef90a6cab6448ab130fab86a69e90f30a029ac141c7e81e3e7d9a3" exitCode=0 Oct 30 07:13:20.277654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.277541 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" event={"ID":"e0665c44-138b-419b-89ec-2906d62bf6ee","Type":"ContainerDied","Data":"6f40e6ad24ef90a6cab6448ab130fab86a69e90f30a029ac141c7e81e3e7d9a3"} Oct 30 07:13:20.325152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.325039 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podStartSLOduration=1.325025116 podStartE2EDuration="1.325025116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:20.323993912 +0000 UTC m=+964.014911349" watchObservedRunningTime="2026-10-30 07:13:20.325025116 +0000 UTC m=+964.015942552" Oct 30 07:13:20.341101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.341065 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:20.723149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:20.723062 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:21.465742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.465717 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:21.544007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.543975 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\") pod \"e0665c44-138b-419b-89ec-2906d62bf6ee\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " Oct 30 07:13:21.544194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.544039 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\" (UniqueName: \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\") pod \"e0665c44-138b-419b-89ec-2906d62bf6ee\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " Oct 30 07:13:21.544297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.544276 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\") pod \"e0665c44-138b-419b-89ec-2906d62bf6ee\" (UID: \"e0665c44-138b-419b-89ec-2906d62bf6ee\") " Oct 30 07:13:21.544406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.544387 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca" pod "e0665c44-138b-419b-89ec-2906d62bf6ee" (UID: "e0665c44-138b-419b-89ec-2906d62bf6ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:21.544724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.544702 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:21.546257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.546226 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd" pod "e0665c44-138b-419b-89ec-2906d62bf6ee" (UID: "e0665c44-138b-419b-89ec-2906d62bf6ee"). InnerVolumeSpecName "kube-api-access-nmgzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:21.546257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.546240 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca" pod "e0665c44-138b-419b-89ec-2906d62bf6ee" (UID: "e0665c44-138b-419b-89ec-2906d62bf6ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:21.646423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.646393 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0665c44-138b-419b-89ec-2906d62bf6ee-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:21.646423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.646422 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgzd\" (UniqueName: \"kubernetes.io/projected/e0665c44-138b-419b-89ec-2906d62bf6ee-kube-api-access-nmgzd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:21.725109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.725071 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:21.725261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.725138 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:21.862781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.862704 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p"] Oct 30 07:13:21.863249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.863236 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e0665c44-138b-419b-89ec-2906d62bf6ee" containerName="revoke-keys" Oct 30 07:13:21.863300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.863250 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0665c44-138b-419b-89ec-2906d62bf6ee" containerName="revoke-keys" Oct 30 07:13:21.863366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.863355 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0665c44-138b-419b-89ec-2906d62bf6ee" containerName="revoke-keys" Oct 30 07:13:21.870065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.870038 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:21.905442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.903524 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p"] Oct 30 07:13:21.951100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.951066 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbdw\" (UniqueName: \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:21.951265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.951220 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:21.951357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:21.951262 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.052699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.052656 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.052885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.052717 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.052885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.052849 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbdw\" (UniqueName: \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.053433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.053410 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.055147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.055123 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.090664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.070625 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:22.107526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.107134 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbdw\" (UniqueName: \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\") pod \"maas-api-revoke-sub-c068d358-clk4p\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.218704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.218597 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:22.317078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.314497 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" event={"ID":"e0665c44-138b-419b-89ec-2906d62bf6ee","Type":"ContainerDied","Data":"fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423"} Oct 30 07:13:22.317078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.314565 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb5cf531813ba80ea1b470d1ed4069b9ddc98aa3416e034fa28f8ff6e8935423" Oct 30 07:13:22.317078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.314591 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl" Oct 30 07:13:22.368359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.368321 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:13:22.407783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.406883 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:13:22.407783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.407017 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.456987 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457024 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7bc\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457191 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457229 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457318 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457373 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457468 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457530 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.457615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.457603 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559192 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559239 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559326 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559380 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559399 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7bc\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559491 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559517 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559564 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559688 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.559852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559740 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.561053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.559984 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.561053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.560119 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.561053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.560503 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.561196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.561116 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.562629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.562608 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.562748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.562728 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.564858 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:22.564831 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c271f8_333c_44d8_8633_259d55bb6347.slice/crio-31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3 WatchSource:0}: Error finding container 31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3: Status 404 returned error can't find the container with id 31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3 Oct 30 07:13:22.565929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.565909 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p"] Oct 30 07:13:22.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.600028 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.607672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.607602 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7bc\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.722944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.722905 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:22.723101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.723010 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:22.723101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:22.723058 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:23.126315 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:23.126284 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210419a8_cabf_4139_b373_921abcf0141d.slice/crio-7db1a34f2e23d6392945a17f5920e015aabcaec41fa27f99b1a377e05306ffb8 WatchSource:0}: Error finding container 7db1a34f2e23d6392945a17f5920e015aabcaec41fa27f99b1a377e05306ffb8: Status 404 returned error can't find the container with id 7db1a34f2e23d6392945a17f5920e015aabcaec41fa27f99b1a377e05306ffb8 Oct 30 07:13:23.127251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.127225 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:13:23.128808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.128775 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.128885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.128851 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.128935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.128885 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:23.324953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.324849 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" event={"ID":"210419a8-cabf-4139-b373-921abcf0141d","Type":"ContainerStarted","Data":"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387"} Oct 30 07:13:23.324953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.324910 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" event={"ID":"210419a8-cabf-4139-b373-921abcf0141d","Type":"ContainerStarted","Data":"7db1a34f2e23d6392945a17f5920e015aabcaec41fa27f99b1a377e05306ffb8"} Oct 30 07:13:23.326568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.326539 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" event={"ID":"e4c271f8-333c-44d8-8633-259d55bb6347","Type":"ContainerStarted","Data":"c672780c90c7273013d8c734e4828f86a28cd78c5bb6bbfe46263a284c495cca"} Oct 30 07:13:23.326725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.326602 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" event={"ID":"e4c271f8-333c-44d8-8633-259d55bb6347","Type":"ContainerStarted","Data":"31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3"} Oct 30 07:13:23.403403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.403284 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podStartSLOduration=1.403262939 podStartE2EDuration="1.403262939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:23.378115614 +0000 UTC m=+967.069033064" watchObservedRunningTime="2026-10-30 07:13:23.403262939 +0000 UTC m=+967.094180377" Oct 30 07:13:23.428388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.428332 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" podStartSLOduration=2.42831904 podStartE2EDuration="2.42831904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:23.425712802 +0000 UTC m=+967.116630240" watchObservedRunningTime="2026-10-30 07:13:23.42831904 +0000 UTC m=+967.119236476" Oct 30 07:13:23.726661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.724015 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:23.726661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.724492 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:23.726661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.724575 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:23.993722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:23.992871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:24.334829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.334799 5894 generic.go:356] "Generic (PLEG): container finished" podID="e4c271f8-333c-44d8-8633-259d55bb6347" containerID="c672780c90c7273013d8c734e4828f86a28cd78c5bb6bbfe46263a284c495cca" exitCode=0 Oct 30 07:13:24.334986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.334890 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" event={"ID":"e4c271f8-333c-44d8-8633-259d55bb6347","Type":"ContainerDied","Data":"c672780c90c7273013d8c734e4828f86a28cd78c5bb6bbfe46263a284c495cca"} Oct 30 07:13:24.724259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.723741 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:24.724259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.723804 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:24.725541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.725513 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:24.725717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:24.725560 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:25.665137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.665110 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:25.723658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.723561 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:25.723658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.723617 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:25.724784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.724245 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:25.724784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.724286 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:25.849339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.849297 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\") pod \"e4c271f8-333c-44d8-8633-259d55bb6347\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " Oct 30 07:13:25.849531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.849364 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\") pod \"e4c271f8-333c-44d8-8633-259d55bb6347\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " Oct 30 07:13:25.849531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.849514 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\" (UniqueName: \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\") pod \"e4c271f8-333c-44d8-8633-259d55bb6347\" (UID: \"e4c271f8-333c-44d8-8633-259d55bb6347\") " Oct 30 07:13:25.849716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.849688 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca" pod "e4c271f8-333c-44d8-8633-259d55bb6347" (UID: "e4c271f8-333c-44d8-8633-259d55bb6347"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:25.850160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.850140 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:25.851826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.851795 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca" pod "e4c271f8-333c-44d8-8633-259d55bb6347" (UID: "e4c271f8-333c-44d8-8633-259d55bb6347"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:25.851963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.851943 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw" pod "e4c271f8-333c-44d8-8633-259d55bb6347" (UID: "e4c271f8-333c-44d8-8633-259d55bb6347"). InnerVolumeSpecName "kube-api-access-8tbdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:25.860611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.860590 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:25.952439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.952312 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4c271f8-333c-44d8-8633-259d55bb6347-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:25.952439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:25.952343 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbdw\" (UniqueName: \"kubernetes.io/projected/e4c271f8-333c-44d8-8633-259d55bb6347-kube-api-access-8tbdw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:26.026165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.026093 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:26.351655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.351616 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" Oct 30 07:13:26.351843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.351662 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p" event={"ID":"e4c271f8-333c-44d8-8633-259d55bb6347","Type":"ContainerDied","Data":"31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3"} Oct 30 07:13:26.351843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.351721 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e3aedc3ea29b373030ea1532635de6da91a4a14e31cf586727c369e60c05b3" Oct 30 07:13:26.723250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.723151 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 07:13:26.723250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.723210 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 07:13:26.723250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.723161 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:26.723911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.723280 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:26.949308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:26.949272 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:27.067254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.067212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:27.629246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.629207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:27.723309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.723271 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:27.723702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.723326 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:27.727356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.727331 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:27.727802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.727780 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:27.728503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:27.728486 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:13:28.724051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:28.724016 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:28.724422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:28.724069 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:29.316679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:29.316622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:29.430999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:29.430959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:29.723411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:29.723321 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 07:13:29.723411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:29.723387 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 07:13:29.910159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:29.910126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:30.042923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:30.028126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:30.785468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:30.785429 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:30.786794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:30.786768 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:30.787260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:30.787237 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:13:31.239619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.239539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:31.818764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.818723 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g"] Oct 30 07:13:31.819357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.819337 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e4c271f8-333c-44d8-8633-259d55bb6347" containerName="revoke-keys" Oct 30 07:13:31.819452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.819358 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c271f8-333c-44d8-8633-259d55bb6347" containerName="revoke-keys" Oct 30 07:13:31.819534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.819515 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c271f8-333c-44d8-8633-259d55bb6347" containerName="revoke-keys" Oct 30 07:13:31.823103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.823080 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.845428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.845391 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g"] Oct 30 07:13:31.854807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.854773 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:13:31.854807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.854781 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:13:31.855030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.854790 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:13:31.855030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.854831 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:13:31.858941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.858911 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.859046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.859028 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2gd6\" (UniqueName: \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.859144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.859119 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.960477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.960440 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.960715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.960495 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2gd6\" (UniqueName: \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.960715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.960524 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.961186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.961157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.963118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.963084 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:31.986266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:31.986232 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2gd6\" (UniqueName: \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\") pod \"maas-api-revoke-sub-442b05d7-tx29g\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:32.137892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:32.137805 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:32.466109 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:32.466080 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb369d465_6781_465b_a0e2_ceb1b5aef310.slice/crio-f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d WatchSource:0}: Error finding container f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d: Status 404 returned error can't find the container with id f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d Oct 30 07:13:32.466585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:32.466558 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g"] Oct 30 07:13:32.840436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:32.840399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:33.418548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.418502 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" event={"ID":"b369d465-6781-465b-a0e2-ceb1b5aef310","Type":"ContainerStarted","Data":"950d76af2c628cc2a8f7cfc8b9da38c8f35b96b90e40ca2d238fea9ae6e8741e"} Oct 30 07:13:33.418694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.418577 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" event={"ID":"b369d465-6781-465b-a0e2-ceb1b5aef310","Type":"ContainerStarted","Data":"f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d"} Oct 30 07:13:33.463733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.463672 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" podStartSLOduration=2.463657377 podStartE2EDuration="2.463657377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:33.462838306 +0000 UTC m=+977.153755744" watchObservedRunningTime="2026-10-30 07:13:33.463657377 +0000 UTC m=+977.154574813" Oct 30 07:13:33.557098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.557065 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:33.712666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.712566 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:33.855654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.855587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:33.992478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:33.992382 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:34.177329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.177301 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 07:13:34.187013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.186984 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.262504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.262416 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2f845b\"/\"distinct-2-w3-2f845b-kserve-self-signed-certs\"" Oct 30 07:13:34.328763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.316410 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.342682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.329078 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.384494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.384459 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmgm\" (UniqueName: \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.384854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.384828 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.385057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.385036 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.391659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.385358 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.404879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.321694 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 07:13:34.405120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.345653 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:34.513463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.513373 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.543003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.542964 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.543776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.543752 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.544369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.544349 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmgm\" (UniqueName: \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.544868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.544848 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.545308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.545290 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.545939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.545234 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.546083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.543621 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.546186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.545666 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.562203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.562169 5894 generic.go:356] "Generic (PLEG): container finished" podID="b369d465-6781-465b-a0e2-ceb1b5aef310" containerID="950d76af2c628cc2a8f7cfc8b9da38c8f35b96b90e40ca2d238fea9ae6e8741e" exitCode=0 Oct 30 07:13:34.569327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.569289 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" event={"ID":"b369d465-6781-465b-a0e2-ceb1b5aef310","Type":"ContainerDied","Data":"950d76af2c628cc2a8f7cfc8b9da38c8f35b96b90e40ca2d238fea9ae6e8741e"} Oct 30 07:13:34.592415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.592383 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmgm\" (UniqueName: \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.631852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.631816 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.632491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.632464 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\") pod \"distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:34.845942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:34.845465 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:35.024655 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:13:35.024516 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:13:35.169674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:35.169515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:35.319657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:35.319127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:36.012719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:36.010710 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 07:13:36.065610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:36.060728 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 07:13:36.091849 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:36.091785 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc517bb18_ee3d_4b4c_a49d_911b92922de7.slice/crio-7e70ba07c0eadffade05350233b9b1bf0b64017ddb19f4f324fabcfb641c7753 WatchSource:0}: Error finding container 7e70ba07c0eadffade05350233b9b1bf0b64017ddb19f4f324fabcfb641c7753: Status 404 returned error can't find the container with id 7e70ba07c0eadffade05350233b9b1bf0b64017ddb19f4f324fabcfb641c7753 Oct 30 07:13:36.809660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:36.807917 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:36.849154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:36.849117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:36.983882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:36.983838 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" event={"ID":"c517bb18-ee3d-4b4c-a49d-911b92922de7","Type":"ContainerStarted","Data":"7e70ba07c0eadffade05350233b9b1bf0b64017ddb19f4f324fabcfb641c7753"} Oct 30 07:13:37.011973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.010197 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:37.257481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.257442 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 07:13:37.290805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.267780 5894 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.388726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.280074 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:13:37.389010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.388980 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:13:37.528819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.528535 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:37.587673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.579458 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\" (UniqueName: \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\") pod \"b369d465-6781-465b-a0e2-ceb1b5aef310\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " Oct 30 07:13:37.587673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.579543 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\") pod \"b369d465-6781-465b-a0e2-ceb1b5aef310\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " Oct 30 07:13:37.587673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.579722 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\") pod \"b369d465-6781-465b-a0e2-ceb1b5aef310\" (UID: \"b369d465-6781-465b-a0e2-ceb1b5aef310\") " Oct 30 07:13:37.607293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.597299 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca" pod "b369d465-6781-465b-a0e2-ceb1b5aef310" (UID: "b369d465-6781-465b-a0e2-ceb1b5aef310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:37.624520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.623716 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca" pod "b369d465-6781-465b-a0e2-ceb1b5aef310" (UID: "b369d465-6781-465b-a0e2-ceb1b5aef310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:37.627524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.627493 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6" pod "b369d465-6781-465b-a0e2-ceb1b5aef310" (UID: "b369d465-6781-465b-a0e2-ceb1b5aef310"). InnerVolumeSpecName "kube-api-access-w2gd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:37.709548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.683617 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2gd6\" (UniqueName: \"kubernetes.io/projected/b369d465-6781-465b-a0e2-ceb1b5aef310-kube-api-access-w2gd6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:37.709548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.708357 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:37.709548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.708407 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b369d465-6781-465b-a0e2-ceb1b5aef310-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:37.851925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:37.851834 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:38.004136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.003697 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" event={"ID":"c517bb18-ee3d-4b4c-a49d-911b92922de7","Type":"ContainerStarted","Data":"de5858fd3890cac142125c2f2adc61f0bf7916693770ebff209acb0f7af7c6fc"} Oct 30 07:13:38.008720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.008685 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" event={"ID":"b369d465-6781-465b-a0e2-ceb1b5aef310","Type":"ContainerDied","Data":"f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d"} Oct 30 07:13:38.008895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.008759 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03e2d99984c54dc396e5681ae4a9bc4f07e143f92e4969dcaa39a80865bc03d" Oct 30 07:13:38.008895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.008819 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g" Oct 30 07:13:38.054470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.054416 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" podStartSLOduration=4.054397971 podStartE2EDuration="4.054397971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:38.051529687 +0000 UTC m=+981.742447125" watchObservedRunningTime="2026-10-30 07:13:38.054397971 +0000 UTC m=+981.745315402" Oct 30 07:13:38.170077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.169966 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:38.708426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.708383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:38.846881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:38.846847 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:40.392268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:40.392237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:41.817248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:41.817195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:42.101617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.101544 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz"] Oct 30 07:13:42.102114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.102100 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b369d465-6781-465b-a0e2-ceb1b5aef310" containerName="revoke-keys" Oct 30 07:13:42.102163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.102116 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b369d465-6781-465b-a0e2-ceb1b5aef310" containerName="revoke-keys" Oct 30 07:13:42.102255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.102246 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b369d465-6781-465b-a0e2-ceb1b5aef310" containerName="revoke-keys" Oct 30 07:13:42.105509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.105494 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.123011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.122976 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz"] Oct 30 07:13:42.133738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.133712 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:13:42.133899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.133747 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:13:42.133899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.133748 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:13:42.142761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.142739 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:13:42.202966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.202926 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq"] Oct 30 07:13:42.207223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.207204 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.225247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.225218 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq"] Oct 30 07:13:42.270286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.270253 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.270503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.270479 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56jtd\" (UniqueName: \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.270594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.270527 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.372364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372266 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.372364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372324 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrclz\" (UniqueName: \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.372578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372433 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.372654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372592 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56jtd\" (UniqueName: \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.372712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372624 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.372712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.372707 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.373374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.373345 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.374969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.374948 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.408258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.408227 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56jtd\" (UniqueName: \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\") pod \"maas-api-revoke-sub-a600f1a8-kxrgz\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.420111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.420080 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:42.474751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.474713 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.474907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.474826 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.474907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.474875 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrclz\" (UniqueName: \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.476794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.476174 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.479849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.478294 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.500771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.500739 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrclz\" (UniqueName: \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\") pod \"maas-api-revoke-sub-449b92fb-98mfq\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.522912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.522879 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:42.724472 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:42.724428 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f768d86_8792_45f6_83d3_fb24034134c1.slice/crio-eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b WatchSource:0}: Error finding container eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b: Status 404 returned error can't find the container with id eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b Oct 30 07:13:42.725810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.725786 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz"] Oct 30 07:13:42.833009 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:42.832978 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9295b961_ff2f_47d6_a21f_094f58f7061e.slice/crio-e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2 WatchSource:0}: Error finding container e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2: Status 404 returned error can't find the container with id e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2 Oct 30 07:13:42.833666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:42.833629 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq"] Oct 30 07:13:43.045192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:43.045153 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" event={"ID":"3f768d86-8792-45f6-83d3-fb24034134c1","Type":"ContainerStarted","Data":"eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b"} Oct 30 07:13:43.046519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:43.046494 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" event={"ID":"9295b961-ff2f-47d6-a21f-094f58f7061e","Type":"ContainerStarted","Data":"e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2"} Oct 30 07:13:43.412740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:43.412659 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:44.055761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.055725 5894 generic.go:356] "Generic (PLEG): container finished" podID="9295b961-ff2f-47d6-a21f-094f58f7061e" containerID="10a5a01d3a5dfed2d2191383822be943160b8498875a9c641ea4ebc677faba15" exitCode=0 Oct 30 07:13:44.056223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.055805 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" event={"ID":"9295b961-ff2f-47d6-a21f-094f58f7061e","Type":"ContainerDied","Data":"10a5a01d3a5dfed2d2191383822be943160b8498875a9c641ea4ebc677faba15"} Oct 30 07:13:44.057569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.057537 5894 generic.go:356] "Generic (PLEG): container finished" podID="3f768d86-8792-45f6-83d3-fb24034134c1" containerID="8f79ac2f6519e7b4bb400a74dbb8a09c50045ed3082617a084e2769b30c6f40b" exitCode=0 Oct 30 07:13:44.057718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.057607 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" event={"ID":"3f768d86-8792-45f6-83d3-fb24034134c1","Type":"ContainerDied","Data":"8f79ac2f6519e7b4bb400a74dbb8a09c50045ed3082617a084e2769b30c6f40b"} Oct 30 07:13:44.847419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.847376 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:44.860373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:44.860343 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:45.085705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.085677 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 07:13:45.101256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.100935 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:45.253174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.253004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:45.255105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.255082 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:45.281095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.281066 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:45.411839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.411758 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\") pod \"3f768d86-8792-45f6-83d3-fb24034134c1\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " Oct 30 07:13:45.411988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.411863 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\") pod \"9295b961-ff2f-47d6-a21f-094f58f7061e\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " Oct 30 07:13:45.411988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.411962 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\" (UniqueName: \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\") pod \"3f768d86-8792-45f6-83d3-fb24034134c1\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " Oct 30 07:13:45.412065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.411997 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\") pod \"9295b961-ff2f-47d6-a21f-094f58f7061e\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " Oct 30 07:13:45.412065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412039 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\" (UniqueName: \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\") pod \"9295b961-ff2f-47d6-a21f-094f58f7061e\" (UID: \"9295b961-ff2f-47d6-a21f-094f58f7061e\") " Oct 30 07:13:45.412161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412112 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\") pod \"3f768d86-8792-45f6-83d3-fb24034134c1\" (UID: \"3f768d86-8792-45f6-83d3-fb24034134c1\") " Oct 30 07:13:45.412222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412162 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca" pod "3f768d86-8792-45f6-83d3-fb24034134c1" (UID: "3f768d86-8792-45f6-83d3-fb24034134c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:45.412874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412595 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca" pod "9295b961-ff2f-47d6-a21f-094f58f7061e" (UID: "9295b961-ff2f-47d6-a21f-094f58f7061e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:13:45.412874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412796 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:45.412874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.412813 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:45.414372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.414346 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca" pod "3f768d86-8792-45f6-83d3-fb24034134c1" (UID: "3f768d86-8792-45f6-83d3-fb24034134c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:45.414491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.414372 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd" pod "3f768d86-8792-45f6-83d3-fb24034134c1" (UID: "3f768d86-8792-45f6-83d3-fb24034134c1"). InnerVolumeSpecName "kube-api-access-56jtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:45.414491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.414413 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca" pod "9295b961-ff2f-47d6-a21f-094f58f7061e" (UID: "9295b961-ff2f-47d6-a21f-094f58f7061e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:13:45.414491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.414445 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz" pod "9295b961-ff2f-47d6-a21f-094f58f7061e" (UID: "9295b961-ff2f-47d6-a21f-094f58f7061e"). InnerVolumeSpecName "kube-api-access-vrclz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:13:45.514293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.514256 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56jtd\" (UniqueName: \"kubernetes.io/projected/3f768d86-8792-45f6-83d3-fb24034134c1-kube-api-access-56jtd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:45.514293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.514286 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrclz\" (UniqueName: \"kubernetes.io/projected/9295b961-ff2f-47d6-a21f-094f58f7061e-kube-api-access-vrclz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:45.514293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.514298 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f768d86-8792-45f6-83d3-fb24034134c1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:45.514527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:45.514307 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9295b961-ff2f-47d6-a21f-094f58f7061e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:13:46.075923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.075889 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" Oct 30 07:13:46.075923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.075914 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz" event={"ID":"3f768d86-8792-45f6-83d3-fb24034134c1","Type":"ContainerDied","Data":"eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b"} Oct 30 07:13:46.076203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.075981 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaf4293a4e57739927d3095858e6704ee78f233665a504da2c74be3445afa67b" Oct 30 07:13:46.078759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.078735 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" Oct 30 07:13:46.079099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.079070 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq" event={"ID":"9295b961-ff2f-47d6-a21f-094f58f7061e","Type":"ContainerDied","Data":"e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2"} Oct 30 07:13:46.079237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.079117 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1527a4018727bad235cf8869b9db5f935f4ee73f89bf3aa0ce0d8aff9c333a2" Oct 30 07:13:46.600083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:46.600054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:48.137258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:48.137217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:49.588968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:49.588929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:51.100040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.100006 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:51.304681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.304648 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5"] Oct 30 07:13:51.305231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305202 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3f768d86-8792-45f6-83d3-fb24034134c1" containerName="revoke-keys" Oct 30 07:13:51.305231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305224 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f768d86-8792-45f6-83d3-fb24034134c1" containerName="revoke-keys" Oct 30 07:13:51.305476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305262 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9295b961-ff2f-47d6-a21f-094f58f7061e" containerName="revoke-keys" Oct 30 07:13:51.305476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305268 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9295b961-ff2f-47d6-a21f-094f58f7061e" containerName="revoke-keys" Oct 30 07:13:51.305476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305364 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f768d86-8792-45f6-83d3-fb24034134c1" containerName="revoke-keys" Oct 30 07:13:51.305476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.305372 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9295b961-ff2f-47d6-a21f-094f58f7061e" containerName="revoke-keys" Oct 30 07:13:51.310123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.310103 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.331198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.331169 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5"] Oct 30 07:13:51.341077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.341039 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 07:13:51.341077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.341045 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 07:13:51.341077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.341049 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 07:13:51.341287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.341086 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-725wz\"" Oct 30 07:13:51.392963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.392874 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60bb623-53ca-42c5-af7d-610ee177be92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.392963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.392909 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.392963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.392930 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.393209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.392998 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.393209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.393054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.393209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.393119 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ml6\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-kube-api-access-49ml6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.393369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.393311 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494235 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494289 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ml6\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-kube-api-access-49ml6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494478 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494697 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60bb623-53ca-42c5-af7d-610ee177be92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494731 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494764 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.494864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.494789 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.495585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.495548 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.496997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.496969 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.497086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.496994 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e60bb623-53ca-42c5-af7d-610ee177be92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.497123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.497108 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.497174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.497159 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e60bb623-53ca-42c5-af7d-610ee177be92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.519613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.519569 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.519773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.519756 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ml6\" (UniqueName: \"kubernetes.io/projected/e60bb623-53ca-42c5-af7d-610ee177be92-kube-api-access-49ml6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nf7b5\" (UID: \"e60bb623-53ca-42c5-af7d-610ee177be92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.623951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.623916 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:51.952969 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:51.952883 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60bb623_53ca_42c5_af7d_610ee177be92.slice/crio-1acac707c9a4cf1f263a0ecb0209162561b410f031472aa3fc38c19b0ed231b4 WatchSource:0}: Error finding container 1acac707c9a4cf1f263a0ecb0209162561b410f031472aa3fc38c19b0ed231b4: Status 404 returned error can't find the container with id 1acac707c9a4cf1f263a0ecb0209162561b410f031472aa3fc38c19b0ed231b4 Oct 30 07:13:51.953180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.953138 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5"] Oct 30 07:13:51.955198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:51.955179 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:13:52.123195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:52.123162 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" event={"ID":"e60bb623-53ca-42c5-af7d-610ee177be92","Type":"ContainerStarted","Data":"1acac707c9a4cf1f263a0ecb0209162561b410f031472aa3fc38c19b0ed231b4"} Oct 30 07:13:52.440521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:52.440486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:52.601010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:52.600971 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:53.675696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:53.672780 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:13:54.230048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:54.230007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:56.347692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:56.347225 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:56.347692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:56.347322 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:56.557655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:56.556997 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:57.319672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:57.318334 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" event={"ID":"e60bb623-53ca-42c5-af7d-610ee177be92","Type":"ContainerStarted","Data":"36d12b96ffd7a8afe45d8c97547aa159c02a9c75e607ede0bf4feb3adfa9292b"} Oct 30 07:13:57.323668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:57.321481 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:57.323668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:57.322465 5894 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nf7b5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:13:57.323668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:57.322526 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" podUID="e60bb623-53ca-42c5-af7d-610ee177be92" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:13:57.429517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:57.429448 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" podStartSLOduration=2.038921407 podStartE2EDuration="6.429422366s" totalImagesPullingTime="4.390500959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:57.398284374 +0000 UTC m=+1001.089201814" watchObservedRunningTime="2026-10-30 07:13:57.429422366 +0000 UTC m=+1001.120339805" Oct 30 07:13:58.350803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.350766 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:13:58.359561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.359533 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.420518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.404225 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2f845b-openshift-default-dockercfg-469h2\"" Oct 30 07:13:58.450692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.444695 5894 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nf7b5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:13:58.450692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.444766 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" podUID="e60bb623-53ca-42c5-af7d-610ee177be92" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:13:58.474691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.474025 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513467 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtcd\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513577 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513710 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513739 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513915 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.513962 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.514114 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.514197 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.525678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.514226 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.630748 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.630813 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.630921 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.630971 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.630998 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.631040 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtcd\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.631094 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.631147 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.631175 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.632039 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.632767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.632298 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.642854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.640889 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.648754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.646314 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.648969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.646548 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.650442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.650394 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.652168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.652119 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.658747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.656618 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.667803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.667776 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtcd\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:58.729230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.729188 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:58.760525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:58.760483 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:13:59.153298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.153266 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:13:59.154575 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:13:59.154545 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbdc66c_1f3e_4f70_a003_8cdc6f9af59c.slice/crio-9d9610516a5ab3142a0e326284e9cd56fb0bcf9b149c2f76a33b5f8040d987c3 WatchSource:0}: Error finding container 9d9610516a5ab3142a0e326284e9cd56fb0bcf9b149c2f76a33b5f8040d987c3: Status 404 returned error can't find the container with id 9d9610516a5ab3142a0e326284e9cd56fb0bcf9b149c2f76a33b5f8040d987c3 Oct 30 07:13:59.157185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.157150 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:59.157273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.157218 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:59.157273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.157248 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:13:59.390685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.388893 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" event={"ID":"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c","Type":"ContainerStarted","Data":"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95"} Oct 30 07:13:59.390685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.388952 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" event={"ID":"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c","Type":"ContainerStarted","Data":"9d9610516a5ab3142a0e326284e9cd56fb0bcf9b149c2f76a33b5f8040d987c3"} Oct 30 07:13:59.390685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.390331 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nf7b5" Oct 30 07:13:59.440289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.440191 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podStartSLOduration=1.4401734990000001 podStartE2EDuration="1.440173499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:13:59.43896144 +0000 UTC m=+1003.129878877" watchObservedRunningTime="2026-10-30 07:13:59.440173499 +0000 UTC m=+1003.131090936" Oct 30 07:13:59.537117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.536591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:13:59.760983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:13:59.760893 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:14:00.542279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:00.542240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:00.744962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:00.744918 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:00.763137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:00.763098 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:00.763295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:00.763166 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:01.761915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:01.761876 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:01.762335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:01.761954 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:02.753352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:02.753317 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:02.761318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:02.761292 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:02.761454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:02.761353 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:03.761709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:03.761675 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:03.762121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:03.761726 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:03.896996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:03.896965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:04.444017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.443986 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:04.564997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.564960 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 07:14:04.569351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.569327 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.608083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.608036 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 07:14:04.737085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737004 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737047 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737247 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737280 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737359 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737395 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737501 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737533 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l46wk\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.737688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.737621 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.761776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.761738 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:04.762164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.761794 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:04.830711 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:14:04.830609 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:14:04.839350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839317 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839379 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839422 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839597 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839650 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839701 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839726 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839754 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839806 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839817 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839843 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l46wk\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.839933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.839895 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.840119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.840026 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.840341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.840318 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.842147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.842121 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.843123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.843100 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.866409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.866374 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l46wk\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.866575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.866557 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:04.908512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:04.908300 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:05.294154 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:14:05.294120 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf044fbad_b5ea_4247_9a7b_7169b548759f.slice/crio-e8ff5cf9533254205d46acdebb093087a1edb2b3c08b09e4f711d388f1a4dd13 WatchSource:0}: Error finding container e8ff5cf9533254205d46acdebb093087a1edb2b3c08b09e4f711d388f1a4dd13: Status 404 returned error can't find the container with id e8ff5cf9533254205d46acdebb093087a1edb2b3c08b09e4f711d388f1a4dd13 Oct 30 07:14:05.294929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.294905 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 07:14:05.297188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.297158 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.297249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.297221 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.297288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.297250 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:05.522564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.522515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" event={"ID":"f044fbad-b5ea-4247-9a7b-7169b548759f","Type":"ContainerStarted","Data":"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d"} Oct 30 07:14:05.522789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.522590 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" event={"ID":"f044fbad-b5ea-4247-9a7b-7169b548759f","Type":"ContainerStarted","Data":"e8ff5cf9533254205d46acdebb093087a1edb2b3c08b09e4f711d388f1a4dd13"} Oct 30 07:14:05.580803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.579473 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podStartSLOduration=1.57945611 podStartE2EDuration="1.57945611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:05.57421986 +0000 UTC m=+1009.265137314" watchObservedRunningTime="2026-10-30 07:14:05.57945611 +0000 UTC m=+1009.270373548" Oct 30 07:14:05.761729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.761693 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 07:14:05.761950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.761754 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 07:14:05.908955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:05.908857 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:06.248496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.248381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:06.765924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.765892 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:14:06.766710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.766688 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:14:06.767289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.767264 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:14:06.910868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.910823 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:06.911058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.910886 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:06.924132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:06.924062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:07.909543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:07.909508 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:07.910004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:07.909576 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:07.941998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:07.941961 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:08.909410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:08.909369 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:08.909595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:08.909450 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:09.564367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:09.564327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:09.697474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:09.697438 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:09.909251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:09.909167 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:09.909251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:09.909231 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:10.909162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:10.909108 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:10.909694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:10.909176 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:11.233439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:11.233346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:11.909261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:11.909222 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 07:14:11.909756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:11.909290 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 07:14:12.026744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.026703 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:14:12.033763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.033717 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.062700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.062666 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:14:12.168441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168349 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bgn\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168458 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168541 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168703 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168764 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168856 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168913 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.168978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.168949 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.169025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.169011 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270047 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270152 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270179 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270312 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270342 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270435 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270485 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bgn\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270560 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270604 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.270808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270790 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.271030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.270983 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.271030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.271004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.271251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.271229 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.273213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.273179 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.273325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.273227 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.295337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.295302 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.295474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.295432 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bgn\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\") pod \"maas-default-gateway-openshift-default-6dcf757899-k7lsn\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.359799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.359752 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:12.693099 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:14:12.693064 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abb1e21_3df2_4cd9_af05_870e82de81d3.slice/crio-e49a45467e3329623ba36911aac77c71fed0a45159c052d851f53c2dc9a229a5 WatchSource:0}: Error finding container e49a45467e3329623ba36911aac77c71fed0a45159c052d851f53c2dc9a229a5: Status 404 returned error can't find the container with id e49a45467e3329623ba36911aac77c71fed0a45159c052d851f53c2dc9a229a5 Oct 30 07:14:12.695936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.695899 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.696046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.695981 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.696046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.696026 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:14:12.697608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.697576 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:14:12.834327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.834298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:12.964176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:12.963898 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:13.095026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.094992 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:13.097019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.096995 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:13.109851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.109820 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 07:14:13.604346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.604310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" event={"ID":"1abb1e21-3df2-4cd9-af05-870e82de81d3","Type":"ContainerStarted","Data":"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0"} Oct 30 07:14:13.604514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.604368 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" event={"ID":"1abb1e21-3df2-4cd9-af05-870e82de81d3","Type":"ContainerStarted","Data":"e49a45467e3329623ba36911aac77c71fed0a45159c052d851f53c2dc9a229a5"} Oct 30 07:14:13.655344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:13.655290 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podStartSLOduration=2.655275273 podStartE2EDuration="2.655275273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:13.652558806 +0000 UTC m=+1017.343476245" watchObservedRunningTime="2026-10-30 07:14:13.655275273 +0000 UTC m=+1017.346192710" Oct 30 07:14:14.360859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:14.360826 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:14.362602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:14.362503 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:14.362602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:14.362580 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:14.533113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:14.533067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:15.360788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:15.360755 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:15.360985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:15.360808 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:16.130188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.130150 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:16.360968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.360891 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:16.361385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.361016 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:16.602803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.602770 5894 scope.go:122] "RemoveContainer" containerID="28cbe041f4820839c6e6fa2862ce4cf87c0c4125f3348d8c02eed3efffc2da97" Oct 30 07:14:16.613076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.613054 5894 scope.go:122] "RemoveContainer" containerID="5bba9838580a5bd9310e2d597ccf20336b5040ce1baf15807aad84e5a907444e" Oct 30 07:14:16.622350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.622327 5894 scope.go:122] "RemoveContainer" containerID="dd00323a653de78b5a08585f2596d9edef9aba5b183ccbb9a7613936ff83b051" Oct 30 07:14:16.631769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.631752 5894 scope.go:122] "RemoveContainer" containerID="164aaa60eda99f37a530f5c27c081bc2a47964eb6ee8884782039bcfd7702262" Oct 30 07:14:16.640821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:16.640803 5894 scope.go:122] "RemoveContainer" containerID="b3768383865da31c3209712b3f1a43fc3d0f88b6d1cb773b77e661c901475edc" Oct 30 07:14:17.360946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:17.360915 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:17.361127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:17.360969 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:17.726373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:17.726277 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:18.360521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:18.360487 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:18.360729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:18.360548 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:19.307003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:19.306966 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:19.360812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:19.360776 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:19.361007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:19.360834 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:20.361123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:20.361086 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-k7lsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 07:14:20.361535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:20.361144 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 07:14:20.919450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:20.919412 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:21.621455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:21.621416 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:21.622316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:21.622293 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:21.626847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:21.626816 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:14:22.401169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:22.401126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:23.880167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:23.880135 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:25.403021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:25.402983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:26.161263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:26.161219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:26.184481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:26.184449 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:14:27.094658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:27.093689 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:29.633402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:29.633366 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:31.458009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:31.457976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:33.011946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.011914 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:33.718244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.718210 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g"] Oct 30 07:14:33.722858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.722835 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.741514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.741482 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g"] Oct 30 07:14:33.751348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.751306 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:14:33.751537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.751362 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:14:33.751537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.751426 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:14:33.753603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.753584 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:14:33.869092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.869057 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdl4z\" (UniqueName: \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.869273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.869170 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.869273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.869227 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.970339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.970260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdl4z\" (UniqueName: \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.970339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.970318 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.970541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.970353 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.971107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.971075 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.973079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.973054 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:33.995832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:33.995805 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdl4z\" (UniqueName: \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\") pod \"maas-api-revoke-sub-fa1d30bd-4zj8g\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:34.037765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.037730 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:34.340934 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:14:34.340902 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e772f9_71a0_4de2_9c4b_a549fc649a60.slice/crio-04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7 WatchSource:0}: Error finding container 04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7: Status 404 returned error can't find the container with id 04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7 Oct 30 07:14:34.342024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.342002 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g"] Oct 30 07:14:34.604804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.604768 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:34.828389 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:14:34.828345 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:14:34.876439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.876338 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" event={"ID":"b7e772f9-71a0-4de2-9c4b-a549fc649a60","Type":"ContainerStarted","Data":"f59cc9570588f753c57735f9b3508bb8269b9d82bec68b2abedc2efbe17dbd0a"} Oct 30 07:14:34.876439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.876405 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" event={"ID":"b7e772f9-71a0-4de2-9c4b-a549fc649a60","Type":"ContainerStarted","Data":"04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7"} Oct 30 07:14:34.923464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:34.923402 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" podStartSLOduration=1.9233865890000001 podStartE2EDuration="1.923386589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:14:34.91999782 +0000 UTC m=+1038.610915256" watchObservedRunningTime="2026-10-30 07:14:34.923386589 +0000 UTC m=+1038.614304031" Oct 30 07:14:35.884990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:35.884955 5894 generic.go:356] "Generic (PLEG): container finished" podID="b7e772f9-71a0-4de2-9c4b-a549fc649a60" containerID="f59cc9570588f753c57735f9b3508bb8269b9d82bec68b2abedc2efbe17dbd0a" exitCode=0 Oct 30 07:14:35.885567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:35.885004 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" event={"ID":"b7e772f9-71a0-4de2-9c4b-a549fc649a60","Type":"ContainerDied","Data":"f59cc9570588f753c57735f9b3508bb8269b9d82bec68b2abedc2efbe17dbd0a"} Oct 30 07:14:36.298978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:36.298938 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:37.055692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.055660 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:37.217225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.217128 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\" (UniqueName: \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\") pod \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " Oct 30 07:14:37.217403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.217317 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\") pod \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " Oct 30 07:14:37.217403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.217385 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\") pod \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\" (UID: \"b7e772f9-71a0-4de2-9c4b-a549fc649a60\") " Oct 30 07:14:37.217745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.217721 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca" pod "b7e772f9-71a0-4de2-9c4b-a549fc649a60" (UID: "b7e772f9-71a0-4de2-9c4b-a549fc649a60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:14:37.217997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.217981 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:14:37.219294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.219273 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z" pod "b7e772f9-71a0-4de2-9c4b-a549fc649a60" (UID: "b7e772f9-71a0-4de2-9c4b-a549fc649a60"). InnerVolumeSpecName "kube-api-access-pdl4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:14:37.219442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.219423 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca" pod "b7e772f9-71a0-4de2-9c4b-a549fc649a60" (UID: "b7e772f9-71a0-4de2-9c4b-a549fc649a60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:14:37.319480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.319441 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7e772f9-71a0-4de2-9c4b-a549fc649a60-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:14:37.319480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.319489 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdl4z\" (UniqueName: \"kubernetes.io/projected/b7e772f9-71a0-4de2-9c4b-a549fc649a60-kube-api-access-pdl4z\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:14:37.804853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.804821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:37.901353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.901319 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" event={"ID":"b7e772f9-71a0-4de2-9c4b-a549fc649a60","Type":"ContainerDied","Data":"04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7"} Oct 30 07:14:37.901531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.901375 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04e221ac9a2d7bc4a26590f2f19aec966f5cf803c428e30c8054f9ca56dea9b7" Oct 30 07:14:37.901531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:37.901332 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g" Oct 30 07:14:39.300825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:39.300794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:40.804656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:40.804588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:42.378767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:42.378718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:43.886390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:43.886345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:45.415926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:45.415895 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:47.210784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:47.210743 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:48.706041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:48.706004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:50.220138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:50.220104 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:51.765534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:51.765480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:53.304225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:53.304189 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:54.810463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:54.810424 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:55.365662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:55.365613 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:14:56.474265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:56.474223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:57.963244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:57.963211 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:59.589118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:59.589086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:14:59.947307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:14:59.947211 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:00.071707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.071677 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:00.167365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.167324 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:00.168551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.168521 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b7e772f9-71a0-4de2-9c4b-a549fc649a60" containerName="revoke-keys" Oct 30 07:15:00.168551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.168549 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7e772f9-71a0-4de2-9c4b-a549fc649a60" containerName="revoke-keys" Oct 30 07:15:00.168747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.168726 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7e772f9-71a0-4de2-9c4b-a549fc649a60" containerName="revoke-keys" Oct 30 07:15:00.173762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.173738 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.174713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.174692 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:00.178808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.178784 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:00.178966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.178942 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.183218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.183099 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.184028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.184004 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:00.188763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.188737 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.191850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.191810 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:15:00.195508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.195473 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:15:00.196099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.196072 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:00.201335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.201267 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:00.203977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.203950 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:15:00.214089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.210774 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:00.218777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.218374 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.221710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.221682 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:00.227700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.227422 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:00.244483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.244313 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:00.339770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.339733 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.339957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.339786 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvcq\" (UniqueName: \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.339957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.339861 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgff\" (UniqueName: \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.340050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.339957 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cc7t\" (UniqueName: \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.340050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340011 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.340118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340051 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscvm\" (UniqueName: \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.340151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340140 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.340185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340175 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgjbg\" (UniqueName: \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.340237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340205 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.340416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.340388 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.441188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441153 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.441361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441212 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscvm\" (UniqueName: \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.441361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441288 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.441361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441331 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgjbg\" (UniqueName: \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.441477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441370 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.441530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441494 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.441594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441574 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.441672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441628 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvcq\" (UniqueName: \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.441813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.441792 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgff\" (UniqueName: \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.442143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442020 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.442143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442029 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cc7t\" (UniqueName: \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.442143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442082 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.442336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442185 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.442336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442281 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.442421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.442403 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.466934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.466865 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvcq\" (UniqueName: \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.467065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.466998 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cc7t\" (UniqueName: \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\") pod \"maas-api-key-cleanup-29889075-hblnq\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.467065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.467001 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgjbg\" (UniqueName: \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\") pod \"maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.467140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.467063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgff\" (UniqueName: \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.467435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.467419 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscvm\" (UniqueName: \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.493498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.493467 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:00.512360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.512333 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:00.527990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.527772 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:00.548683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.546837 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:00.554139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.554109 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:00.821025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.820155 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:00.825256 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:00.825214 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1d2c98_252d_4806_a617_3209dd7a6c0a.slice/crio-fe8d9be207153d743f577ae0b6117379294b58c808264c19d49e95fe907b5676 WatchSource:0}: Error finding container fe8d9be207153d743f577ae0b6117379294b58c808264c19d49e95fe907b5676: Status 404 returned error can't find the container with id fe8d9be207153d743f577ae0b6117379294b58c808264c19d49e95fe907b5676 Oct 30 07:15:00.851058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.842414 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:00.851058 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:00.843070 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cbec18_b8bb_4311_a41f_9714d8950ad0.slice/crio-c0996de1f514cc22e98f5faed9032d599e6f322907372ca789bbcd813f4cd28a WatchSource:0}: Error finding container c0996de1f514cc22e98f5faed9032d599e6f322907372ca789bbcd813f4cd28a: Status 404 returned error can't find the container with id c0996de1f514cc22e98f5faed9032d599e6f322907372ca789bbcd813f4cd28a Oct 30 07:15:00.869663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.869619 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:00.910293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.910257 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:00.935486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:00.935453 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:00.976793 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:00.976752 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139311fd_052e_4fc1_9d15_ff7679d34937.slice/crio-5fc9a8a7e21fe1bc8876e7e72e63bbc7e4a722ed9a3dbaf0dd4ef8cb85285e69 WatchSource:0}: Error finding container 5fc9a8a7e21fe1bc8876e7e72e63bbc7e4a722ed9a3dbaf0dd4ef8cb85285e69: Status 404 returned error can't find the container with id 5fc9a8a7e21fe1bc8876e7e72e63bbc7e4a722ed9a3dbaf0dd4ef8cb85285e69 Oct 30 07:15:00.987180 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:00.987137 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2059b07_d8b1_47d2_9071_9e3dd6d937ea.slice/crio-51614d4c671bd0cfbf8f2b2157f53f94ce752bb338e0757494889bb94fca9005 WatchSource:0}: Error finding container 51614d4c671bd0cfbf8f2b2157f53f94ce752bb338e0757494889bb94fca9005: Status 404 returned error can't find the container with id 51614d4c671bd0cfbf8f2b2157f53f94ce752bb338e0757494889bb94fca9005 Oct 30 07:15:01.100600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.100563 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerStarted","Data":"fe8d9be207153d743f577ae0b6117379294b58c808264c19d49e95fe907b5676"} Oct 30 07:15:01.101922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.101897 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerStarted","Data":"7c3e84a497a9d1cb75c784ad09997b8d224d0d18dbf308970790e96bac3ef2c0"} Oct 30 07:15:01.103137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.103116 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" event={"ID":"b6cbec18-b8bb-4311-a41f-9714d8950ad0","Type":"ContainerStarted","Data":"c0996de1f514cc22e98f5faed9032d599e6f322907372ca789bbcd813f4cd28a"} Oct 30 07:15:01.104282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.104259 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerStarted","Data":"51614d4c671bd0cfbf8f2b2157f53f94ce752bb338e0757494889bb94fca9005"} Oct 30 07:15:01.105533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.105514 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerStarted","Data":"5fc9a8a7e21fe1bc8876e7e72e63bbc7e4a722ed9a3dbaf0dd4ef8cb85285e69"} Oct 30 07:15:01.107421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:01.107394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:02.114890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.114851 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerStarted","Data":"8113f7ccbf1ef190c48459c34465e776e853c5966bd07d304ad108ebd83cf8e7"} Oct 30 07:15:02.116712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.116681 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerStarted","Data":"952d63590dba6bf50d46fca855a5ecaaee48c534e8aba456b3f150df52ad1f68"} Oct 30 07:15:02.118444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.118418 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerID="eb072247cfc3ce4bd8e646336ce8d570783fa9cf08cd594e45eae0ae9fb88a0f" exitCode=60 Oct 30 07:15:02.118585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.118494 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerDied","Data":"eb072247cfc3ce4bd8e646336ce8d570783fa9cf08cd594e45eae0ae9fb88a0f"} Oct 30 07:15:02.118727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.118712 5894 scope.go:122] "RemoveContainer" containerID="eb072247cfc3ce4bd8e646336ce8d570783fa9cf08cd594e45eae0ae9fb88a0f" Oct 30 07:15:02.120171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.120144 5894 generic.go:356] "Generic (PLEG): container finished" podID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerID="6d777a0a49a6e5f597927ff2523a67e834671bd76f7d24a206805ec720adabdd" exitCode=60 Oct 30 07:15:02.120385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.120171 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerDied","Data":"6d777a0a49a6e5f597927ff2523a67e834671bd76f7d24a206805ec720adabdd"} Oct 30 07:15:02.120385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.120382 5894 scope.go:122] "RemoveContainer" containerID="6d777a0a49a6e5f597927ff2523a67e834671bd76f7d24a206805ec720adabdd" Oct 30 07:15:02.122218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.122120 5894 generic.go:356] "Generic (PLEG): container finished" podID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerID="406d66454bf6fe429459238682f52ce0badc6dd68641f7864fbd59705ce14597" exitCode=60 Oct 30 07:15:02.122218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.122175 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" event={"ID":"b6cbec18-b8bb-4311-a41f-9714d8950ad0","Type":"ContainerDied","Data":"406d66454bf6fe429459238682f52ce0badc6dd68641f7864fbd59705ce14597"} Oct 30 07:15:02.122416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.122400 5894 scope.go:122] "RemoveContainer" containerID="406d66454bf6fe429459238682f52ce0badc6dd68641f7864fbd59705ce14597" Oct 30 07:15:02.161415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.161351 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" podStartSLOduration=2.161332464 podStartE2EDuration="2.161332464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.159147939 +0000 UTC m=+1065.850065377" watchObservedRunningTime="2026-10-30 07:15:02.161332464 +0000 UTC m=+1065.852249902" Oct 30 07:15:02.247545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.247430 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" podStartSLOduration=2.24741528 podStartE2EDuration="2.24741528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:02.24515066 +0000 UTC m=+1065.936068096" watchObservedRunningTime="2026-10-30 07:15:02.24741528 +0000 UTC m=+1065.938332753" Oct 30 07:15:02.595571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:02.595540 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:03.132346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.132309 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerStarted","Data":"0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6"} Oct 30 07:15:03.134578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.134545 5894 generic.go:356] "Generic (PLEG): container finished" podID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerID="24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348" exitCode=60 Oct 30 07:15:03.134752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.134601 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerDied","Data":"24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348"} Oct 30 07:15:03.134752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.134692 5894 scope.go:122] "RemoveContainer" containerID="6d777a0a49a6e5f597927ff2523a67e834671bd76f7d24a206805ec720adabdd" Oct 30 07:15:03.134976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.134958 5894 scope.go:122] "RemoveContainer" containerID="24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348" Oct 30 07:15:03.135354 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:03.135280 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr_odh-ai-gateway-infra(b7075834-d010-4d8e-9241-d3b17f6c4cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" Oct 30 07:15:03.136972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.136949 5894 generic.go:356] "Generic (PLEG): container finished" podID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerID="3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621" exitCode=60 Oct 30 07:15:03.137058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.137032 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" event={"ID":"b6cbec18-b8bb-4311-a41f-9714d8950ad0","Type":"ContainerDied","Data":"3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621"} Oct 30 07:15:03.137232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.137217 5894 scope.go:122] "RemoveContainer" containerID="3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621" Oct 30 07:15:03.137481 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:03.137462 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm_odh-ai-gateway-infra(b6cbec18-b8bb-4311-a41f-9714d8950ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" Oct 30 07:15:03.138549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.138531 5894 generic.go:356] "Generic (PLEG): container finished" podID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerID="8113f7ccbf1ef190c48459c34465e776e853c5966bd07d304ad108ebd83cf8e7" exitCode=60 Oct 30 07:15:03.138682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.138610 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerDied","Data":"8113f7ccbf1ef190c48459c34465e776e853c5966bd07d304ad108ebd83cf8e7"} Oct 30 07:15:03.138820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.138802 5894 scope.go:122] "RemoveContainer" containerID="8113f7ccbf1ef190c48459c34465e776e853c5966bd07d304ad108ebd83cf8e7" Oct 30 07:15:03.140396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.140372 5894 generic.go:356] "Generic (PLEG): container finished" podID="139311fd-052e-4fc1-9d15-ff7679d34937" containerID="952d63590dba6bf50d46fca855a5ecaaee48c534e8aba456b3f150df52ad1f68" exitCode=60 Oct 30 07:15:03.140493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.140456 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerDied","Data":"952d63590dba6bf50d46fca855a5ecaaee48c534e8aba456b3f150df52ad1f68"} Oct 30 07:15:03.140648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.140616 5894 scope.go:122] "RemoveContainer" containerID="952d63590dba6bf50d46fca855a5ecaaee48c534e8aba456b3f150df52ad1f68" Oct 30 07:15:03.192926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.192865 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" podStartSLOduration=3.192847238 podStartE2EDuration="3.192847238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:03.187084824 +0000 UTC m=+1066.878002262" watchObservedRunningTime="2026-10-30 07:15:03.192847238 +0000 UTC m=+1066.883764675" Oct 30 07:15:03.251477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:03.251452 5894 scope.go:122] "RemoveContainer" containerID="406d66454bf6fe429459238682f52ce0badc6dd68641f7864fbd59705ce14597" Oct 30 07:15:04.152812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.152783 5894 scope.go:122] "RemoveContainer" containerID="3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621" Oct 30 07:15:04.153269 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.153157 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm_odh-ai-gateway-infra(b6cbec18-b8bb-4311-a41f-9714d8950ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" Oct 30 07:15:04.154853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.154825 5894 generic.go:356] "Generic (PLEG): container finished" podID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerID="3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4" exitCode=60 Oct 30 07:15:04.154999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.154878 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerDied","Data":"3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4"} Oct 30 07:15:04.154999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.154940 5894 scope.go:122] "RemoveContainer" containerID="8113f7ccbf1ef190c48459c34465e776e853c5966bd07d304ad108ebd83cf8e7" Oct 30 07:15:04.155226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.155201 5894 scope.go:122] "RemoveContainer" containerID="3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4" Oct 30 07:15:04.155560 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.155535 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97_odh-ai-gateway-infra(d2059b07-d8b1-47d2-9071-9e3dd6d937ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" Oct 30 07:15:04.157188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.157164 5894 generic.go:356] "Generic (PLEG): container finished" podID="139311fd-052e-4fc1-9d15-ff7679d34937" containerID="783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa" exitCode=60 Oct 30 07:15:04.157286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.157216 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerDied","Data":"783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa"} Oct 30 07:15:04.157544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.157524 5894 scope.go:122] "RemoveContainer" containerID="783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa" Oct 30 07:15:04.157911 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.157870 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hblnq_odh-ai-gateway-infra(139311fd-052e-4fc1-9d15-ff7679d34937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" Oct 30 07:15:04.159442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.159415 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerID="0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6" exitCode=60 Oct 30 07:15:04.159578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.159493 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerDied","Data":"0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6"} Oct 30 07:15:04.159773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.159757 5894 scope.go:122] "RemoveContainer" containerID="0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6" Oct 30 07:15:04.160064 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.160046 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h_odh-ai-gateway-infra(4b1d2c98-252d-4806-a617-3209dd7a6c0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" Oct 30 07:15:04.161489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.161471 5894 scope.go:122] "RemoveContainer" containerID="24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348" Oct 30 07:15:04.161831 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.161810 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr_odh-ai-gateway-infra(b7075834-d010-4d8e-9241-d3b17f6c4cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" Oct 30 07:15:04.168354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.168336 5894 scope.go:122] "RemoveContainer" containerID="952d63590dba6bf50d46fca855a5ecaaee48c534e8aba456b3f150df52ad1f68" Oct 30 07:15:04.175471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.175442 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:04.236865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:04.236830 5894 scope.go:122] "RemoveContainer" containerID="eb072247cfc3ce4bd8e646336ce8d570783fa9cf08cd594e45eae0ae9fb88a0f" Oct 30 07:15:04.829086 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:04.829050 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:15:05.170556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:05.170466 5894 scope.go:122] "RemoveContainer" containerID="3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4" Oct 30 07:15:05.171048 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:05.170837 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97_odh-ai-gateway-infra(d2059b07-d8b1-47d2-9071-9e3dd6d937ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" Oct 30 07:15:05.172331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:05.172312 5894 scope.go:122] "RemoveContainer" containerID="783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa" Oct 30 07:15:05.172685 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:05.172618 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hblnq_odh-ai-gateway-infra(139311fd-052e-4fc1-9d15-ff7679d34937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" Oct 30 07:15:05.173994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:05.173977 5894 scope.go:122] "RemoveContainer" containerID="0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6" Oct 30 07:15:05.174233 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:05.174217 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h_odh-ai-gateway-infra(4b1d2c98-252d-4806-a617-3209dd7a6c0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" Oct 30 07:15:05.680044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:05.679987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:07.124145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:07.124107 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:08.721772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:08.721722 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:10.315344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:10.315307 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:11.913300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:11.913268 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:13.489458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:13.489404 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:14.423913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:14.423881 5894 scope.go:122] "RemoveContainer" containerID="3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621" Oct 30 07:15:14.990779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:14.990697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:15.044921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.044887 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc"] Oct 30 07:15:15.057338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.057299 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f268271d-tt9tc"] Oct 30 07:15:15.250999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.250912 5894 generic.go:356] "Generic (PLEG): container finished" podID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerID="4fa444b1e9aed52b54c16cd4ff198bd4d57b8e4192827ed9a5c9ad4e4aa36fe0" exitCode=60 Oct 30 07:15:15.250999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.250985 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" event={"ID":"b6cbec18-b8bb-4311-a41f-9714d8950ad0","Type":"ContainerDied","Data":"4fa444b1e9aed52b54c16cd4ff198bd4d57b8e4192827ed9a5c9ad4e4aa36fe0"} Oct 30 07:15:15.251194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.251076 5894 scope.go:122] "RemoveContainer" containerID="3789639c1d3e62da8347dc8e506a06e445506ba9576c79266eff095b748fc621" Oct 30 07:15:15.251514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.251497 5894 scope.go:122] "RemoveContainer" containerID="4fa444b1e9aed52b54c16cd4ff198bd4d57b8e4192827ed9a5c9ad4e4aa36fe0" Oct 30 07:15:15.251879 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:15.251855 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm_odh-ai-gateway-infra(b6cbec18-b8bb-4311-a41f-9714d8950ad0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" Oct 30 07:15:15.424246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:15.424212 5894 scope.go:122] "RemoveContainer" containerID="24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348" Oct 30 07:15:16.260481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.260441 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerStarted","Data":"9b2fb7971801564be6acd9aa5101c485555aa2f9179d8d5d633e3137616cd1b6"} Oct 30 07:15:16.305413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.305355 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" podStartSLOduration=16.305338159 podStartE2EDuration="16.305338159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:16.304417556 +0000 UTC m=+1079.995334994" watchObservedRunningTime="2026-10-30 07:15:16.305338159 +0000 UTC m=+1079.996255595" Oct 30 07:15:16.322559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.322528 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:16.433336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.433310 5894 scope.go:122] "RemoveContainer" containerID="783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa" Oct 30 07:15:16.433524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.433498 5894 scope.go:122] "RemoveContainer" containerID="0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6" Oct 30 07:15:16.443233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.443204 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba58660-f309-45f9-9226-4921ecd9153d" path="/var/lib/kubelet/pods/2ba58660-f309-45f9-9226-4921ecd9153d/volumes" Oct 30 07:15:16.504813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.504783 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:16.658342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.658306 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\" (UniqueName: \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\") pod \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " Oct 30 07:15:16.658506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.658494 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\") pod \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\" (UID: \"b6cbec18-b8bb-4311-a41f-9714d8950ad0\") " Oct 30 07:15:16.658882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.658845 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca" pod "b6cbec18-b8bb-4311-a41f-9714d8950ad0" (UID: "b6cbec18-b8bb-4311-a41f-9714d8950ad0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:16.671476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.671439 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg" pod "b6cbec18-b8bb-4311-a41f-9714d8950ad0" (UID: "b6cbec18-b8bb-4311-a41f-9714d8950ad0"). InnerVolumeSpecName "kube-api-access-tgjbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:16.700250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.700063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:16.760512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.760474 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cbec18-b8bb-4311-a41f-9714d8950ad0-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:16.760512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:16.760503 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgjbg\" (UniqueName: \"kubernetes.io/projected/b6cbec18-b8bb-4311-a41f-9714d8950ad0-kube-api-access-tgjbg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:17.282112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.282073 5894 generic.go:356] "Generic (PLEG): container finished" podID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerID="9b2fb7971801564be6acd9aa5101c485555aa2f9179d8d5d633e3137616cd1b6" exitCode=60 Oct 30 07:15:17.282566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.282149 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerDied","Data":"9b2fb7971801564be6acd9aa5101c485555aa2f9179d8d5d633e3137616cd1b6"} Oct 30 07:15:17.282566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.282241 5894 scope.go:122] "RemoveContainer" containerID="24b447675b074aa679207413fa72cf44c0171d122c36586e0ce5deaf0a9bc348" Oct 30 07:15:17.282711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.282665 5894 scope.go:122] "RemoveContainer" containerID="9b2fb7971801564be6acd9aa5101c485555aa2f9179d8d5d633e3137616cd1b6" Oct 30 07:15:17.282995 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:17.282970 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr_odh-ai-gateway-infra(b7075834-d010-4d8e-9241-d3b17f6c4cb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" Oct 30 07:15:17.283802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.283774 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" Oct 30 07:15:17.283913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.283804 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm" event={"ID":"b6cbec18-b8bb-4311-a41f-9714d8950ad0","Type":"ContainerDied","Data":"c0996de1f514cc22e98f5faed9032d599e6f322907372ca789bbcd813f4cd28a"} Oct 30 07:15:17.286316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.286294 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerStarted","Data":"189acde217dd9883f9e0a93022461b24af005b9fb239ffd58053097ffe4a87b0"} Oct 30 07:15:17.289273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.289248 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerStarted","Data":"3ba4aa1c722309ae1b4113544db2de1d5c3867f1c5c5e102364cc14acce89eac"} Oct 30 07:15:17.295772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.295755 5894 scope.go:122] "RemoveContainer" containerID="4fa444b1e9aed52b54c16cd4ff198bd4d57b8e4192827ed9a5c9ad4e4aa36fe0" Oct 30 07:15:17.318628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.318594 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:17.415909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.415874 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:17.420023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.419994 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d0dace8c-29889075-f9scm"] Oct 30 07:15:17.424659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:17.424612 5894 scope.go:122] "RemoveContainer" containerID="3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4" Oct 30 07:15:18.166110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.166071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:18.300115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.300084 5894 generic.go:356] "Generic (PLEG): container finished" podID="139311fd-052e-4fc1-9d15-ff7679d34937" containerID="189acde217dd9883f9e0a93022461b24af005b9fb239ffd58053097ffe4a87b0" exitCode=60 Oct 30 07:15:18.300595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.300257 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerDied","Data":"189acde217dd9883f9e0a93022461b24af005b9fb239ffd58053097ffe4a87b0"} Oct 30 07:15:18.300595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.300318 5894 scope.go:122] "RemoveContainer" containerID="783225a76eddaa012ea7af5e3be8c816ed71d5c22a279ab6fe3313e82b7e29aa" Oct 30 07:15:18.301233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.301212 5894 scope.go:122] "RemoveContainer" containerID="189acde217dd9883f9e0a93022461b24af005b9fb239ffd58053097ffe4a87b0" Oct 30 07:15:18.301697 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:18.301555 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889075-hblnq_odh-ai-gateway-infra(139311fd-052e-4fc1-9d15-ff7679d34937)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" Oct 30 07:15:18.303350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.303329 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerID="3ba4aa1c722309ae1b4113544db2de1d5c3867f1c5c5e102364cc14acce89eac" exitCode=60 Oct 30 07:15:18.303448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.303404 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerDied","Data":"3ba4aa1c722309ae1b4113544db2de1d5c3867f1c5c5e102364cc14acce89eac"} Oct 30 07:15:18.303842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.303822 5894 scope.go:122] "RemoveContainer" containerID="3ba4aa1c722309ae1b4113544db2de1d5c3867f1c5c5e102364cc14acce89eac" Oct 30 07:15:18.304151 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:18.304132 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h_odh-ai-gateway-infra(4b1d2c98-252d-4806-a617-3209dd7a6c0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" Oct 30 07:15:18.312419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.312373 5894 generic.go:356] "Generic (PLEG): container finished" podID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerID="b3cf31f50b1080f5198168fe6919559360cc03860d835fd29b668d553bdb3cf0" exitCode=60 Oct 30 07:15:18.312419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.312410 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerDied","Data":"b3cf31f50b1080f5198168fe6919559360cc03860d835fd29b668d553bdb3cf0"} Oct 30 07:15:18.313021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.313000 5894 scope.go:122] "RemoveContainer" containerID="b3cf31f50b1080f5198168fe6919559360cc03860d835fd29b668d553bdb3cf0" Oct 30 07:15:18.313338 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:18.313320 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97_odh-ai-gateway-infra(d2059b07-d8b1-47d2-9071-9e3dd6d937ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" Oct 30 07:15:18.317893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.317877 5894 scope.go:122] "RemoveContainer" containerID="0d12d62df1eae2fd0cf9a5eae1ae49ab524de8310c0f750ca981a97b864bd5c6" Oct 30 07:15:18.333584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.333563 5894 scope.go:122] "RemoveContainer" containerID="3ea6f29bd93884a4801faa1651ca34b391f1692a7d638dec1656db02c70f5da4" Oct 30 07:15:18.388910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.388875 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:18.436670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.436563 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" path="/var/lib/kubelet/pods/b6cbec18-b8bb-4311-a41f-9714d8950ad0/volumes" Oct 30 07:15:18.474377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.474351 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:18.506107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.506082 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:18.687043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.686944 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\") pod \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " Oct 30 07:15:18.687224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.687046 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\" (UniqueName: \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\") pod \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\" (UID: \"b7075834-d010-4d8e-9241-d3b17f6c4cb8\") " Oct 30 07:15:18.687417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.687386 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca" pod "b7075834-d010-4d8e-9241-d3b17f6c4cb8" (UID: "b7075834-d010-4d8e-9241-d3b17f6c4cb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:18.687621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.687601 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7075834-d010-4d8e-9241-d3b17f6c4cb8-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:18.689163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.689144 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff" pod "b7075834-d010-4d8e-9241-d3b17f6c4cb8" (UID: "b7075834-d010-4d8e-9241-d3b17f6c4cb8"). InnerVolumeSpecName "kube-api-access-wtgff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:18.789452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.789408 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtgff\" (UniqueName: \"kubernetes.io/projected/b7075834-d010-4d8e-9241-d3b17f6c4cb8-kube-api-access-wtgff\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:18.869953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.869915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:18.993069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:18.992965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:19.322506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.322480 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" Oct 30 07:15:19.322953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.322505 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr" event={"ID":"b7075834-d010-4d8e-9241-d3b17f6c4cb8","Type":"ContainerDied","Data":"7c3e84a497a9d1cb75c784ad09997b8d224d0d18dbf308970790e96bac3ef2c0"} Oct 30 07:15:19.322953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.322578 5894 scope.go:122] "RemoveContainer" containerID="9b2fb7971801564be6acd9aa5101c485555aa2f9179d8d5d633e3137616cd1b6" Oct 30 07:15:19.564704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.564660 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:19.605888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.605862 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:19.616782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.616737 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889075-2drlr"] Oct 30 07:15:19.672683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.672467 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:19.675127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.675102 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:19.757833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.757796 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\" (UniqueName: \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\") pod \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " Oct 30 07:15:19.758046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.757904 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\") pod \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\" (UID: \"4b1d2c98-252d-4806-a617-3209dd7a6c0a\") " Oct 30 07:15:19.758046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.757966 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\") pod \"139311fd-052e-4fc1-9d15-ff7679d34937\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " Oct 30 07:15:19.758531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.758273 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\" (UniqueName: \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\") pod \"139311fd-052e-4fc1-9d15-ff7679d34937\" (UID: \"139311fd-052e-4fc1-9d15-ff7679d34937\") " Oct 30 07:15:19.759504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.759447 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca" pod "4b1d2c98-252d-4806-a617-3209dd7a6c0a" (UID: "4b1d2c98-252d-4806-a617-3209dd7a6c0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:19.761074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.759957 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca" pod "139311fd-052e-4fc1-9d15-ff7679d34937" (UID: "139311fd-052e-4fc1-9d15-ff7679d34937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:19.765870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.765841 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm" pod "4b1d2c98-252d-4806-a617-3209dd7a6c0a" (UID: "4b1d2c98-252d-4806-a617-3209dd7a6c0a"). InnerVolumeSpecName "kube-api-access-sscvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:19.765989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.765881 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t" pod "139311fd-052e-4fc1-9d15-ff7679d34937" (UID: "139311fd-052e-4fc1-9d15-ff7679d34937"). InnerVolumeSpecName "kube-api-access-6cc7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:19.793508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.793472 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:19.859700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.859597 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\") pod \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " Oct 30 07:15:19.859700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.859696 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\" (UniqueName: \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\") pod \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\" (UID: \"d2059b07-d8b1-47d2-9071-9e3dd6d937ea\") " Oct 30 07:15:19.860088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860061 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca" pod "d2059b07-d8b1-47d2-9071-9e3dd6d937ea" (UID: "d2059b07-d8b1-47d2-9071-9e3dd6d937ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:19.860411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860388 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.860508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860418 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b1d2c98-252d-4806-a617-3209dd7a6c0a-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.860508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860434 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/139311fd-052e-4fc1-9d15-ff7679d34937-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.860508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860447 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cc7t\" (UniqueName: \"kubernetes.io/projected/139311fd-052e-4fc1-9d15-ff7679d34937-kube-api-access-6cc7t\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.860508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.860462 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscvm\" (UniqueName: \"kubernetes.io/projected/4b1d2c98-252d-4806-a617-3209dd7a6c0a-kube-api-access-sscvm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.862172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.862142 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq" pod "d2059b07-d8b1-47d2-9071-9e3dd6d937ea" (UID: "d2059b07-d8b1-47d2-9071-9e3dd6d937ea"). InnerVolumeSpecName "kube-api-access-6jvcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:19.961770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.961734 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jvcq\" (UniqueName: \"kubernetes.io/projected/d2059b07-d8b1-47d2-9071-9e3dd6d937ea-kube-api-access-6jvcq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:19.968385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:19.968346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:20.339508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.339475 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" Oct 30 07:15:20.339508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.339501 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h" event={"ID":"4b1d2c98-252d-4806-a617-3209dd7a6c0a","Type":"ContainerDied","Data":"fe8d9be207153d743f577ae0b6117379294b58c808264c19d49e95fe907b5676"} Oct 30 07:15:20.340093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.339569 5894 scope.go:122] "RemoveContainer" containerID="3ba4aa1c722309ae1b4113544db2de1d5c3867f1c5c5e102364cc14acce89eac" Oct 30 07:15:20.342198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.342180 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" Oct 30 07:15:20.342346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.342211 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97" event={"ID":"d2059b07-d8b1-47d2-9071-9e3dd6d937ea","Type":"ContainerDied","Data":"51614d4c671bd0cfbf8f2b2157f53f94ce752bb338e0757494889bb94fca9005"} Oct 30 07:15:20.343943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.343919 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" Oct 30 07:15:20.344072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.343934 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq" event={"ID":"139311fd-052e-4fc1-9d15-ff7679d34937","Type":"ContainerDied","Data":"5fc9a8a7e21fe1bc8876e7e72e63bbc7e4a722ed9a3dbaf0dd4ef8cb85285e69"} Oct 30 07:15:20.352936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.352917 5894 scope.go:122] "RemoveContainer" containerID="b3cf31f50b1080f5198168fe6919559360cc03860d835fd29b668d553bdb3cf0" Oct 30 07:15:20.362705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.362678 5894 scope.go:122] "RemoveContainer" containerID="189acde217dd9883f9e0a93022461b24af005b9fb239ffd58053097ffe4a87b0" Oct 30 07:15:20.403591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.403556 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:20.417622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.417587 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889075-vcq5h"] Oct 30 07:15:20.437006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.436777 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" path="/var/lib/kubelet/pods/4b1d2c98-252d-4806-a617-3209dd7a6c0a/volumes" Oct 30 07:15:20.437459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.437438 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" path="/var/lib/kubelet/pods/b7075834-d010-4d8e-9241-d3b17f6c4cb8/volumes" Oct 30 07:15:20.480411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.480377 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:20.484101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.484072 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889075-hblnq"] Oct 30 07:15:20.519958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.519924 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:20.528993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:20.528956 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889075-74w97"] Oct 30 07:15:21.594431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:21.594395 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:22.389435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.389396 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9"] Oct 30 07:15:22.390123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390098 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390121 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390138 5894 cpu_manager.go:414] "containerMap: removing container" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390144 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390153 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390158 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390187 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390192 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390200 5894 cpu_manager.go:414] "containerMap: removing container" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390205 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390212 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390217 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390225 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390230 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390248 5894 cpu_manager.go:414] "containerMap: removing container" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390252 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390262 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390268 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390276 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390280 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390290 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390294 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390305 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390313 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390408 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390415 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390422 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390427 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390433 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390441 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390449 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390455 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390463 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:22.390964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.390470 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:22.396574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.396555 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.413101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.413064 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9"] Oct 30 07:15:22.421949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.421917 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:15:22.422110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.421951 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:15:22.422110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.422042 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:15:22.427322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.427298 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:15:22.434334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.434305 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" path="/var/lib/kubelet/pods/139311fd-052e-4fc1-9d15-ff7679d34937/volumes" Oct 30 07:15:22.434806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.434788 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" path="/var/lib/kubelet/pods/d2059b07-d8b1-47d2-9071-9e3dd6d937ea/volumes" Oct 30 07:15:22.494796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.494758 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jqz\" (UniqueName: \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.494966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.494900 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.495045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.495030 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.596417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.596384 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.596819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.596504 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.596819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.596581 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jqz\" (UniqueName: \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.597169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.597141 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.598773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.598753 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.622138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.622106 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jqz\" (UniqueName: \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\") pod \"maas-api-revoke-sub-ff4d78c4-cxpt9\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:22.711158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:22.711079 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:23.020610 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:23.020577 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82661356_13e8_4a6e_ae5e_78b399bfd05a.slice/crio-3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760 WatchSource:0}: Error finding container 3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760: Status 404 returned error can't find the container with id 3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760 Oct 30 07:15:23.022040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.021970 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9"] Oct 30 07:15:23.038750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.038721 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697"] Oct 30 07:15:23.039327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039312 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:23.039374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039328 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:23.039374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039358 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:23.039374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039363 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:23.039478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039391 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:23.039478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039396 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:23.039550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039495 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="139311fd-052e-4fc1-9d15-ff7679d34937" containerName="cleanup" Oct 30 07:15:23.039550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039501 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2059b07-d8b1-47d2-9071-9e3dd6d937ea" containerName="cleanup" Oct 30 07:15:23.039550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039511 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cbec18-b8bb-4311-a41f-9714d8950ad0" containerName="cleanup" Oct 30 07:15:23.039550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039520 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7075834-d010-4d8e-9241-d3b17f6c4cb8" containerName="cleanup" Oct 30 07:15:23.039550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.039529 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1d2c98-252d-4806-a617-3209dd7a6c0a" containerName="cleanup" Oct 30 07:15:23.044584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.044560 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.062733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.062702 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697"] Oct 30 07:15:23.104802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.104768 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.104981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.104833 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.105163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.105134 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7mm\" (UniqueName: \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.206875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.206837 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.207058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.206985 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.207159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.207147 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7mm\" (UniqueName: \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.207668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.207630 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.209238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.209217 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.224243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.224215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:23.232874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.232849 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7mm\" (UniqueName: \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\") pod \"maas-api-revoke-sub-02b49869-q5697\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.374216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.374141 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" event={"ID":"82661356-13e8-4a6e-ae5e-78b399bfd05a","Type":"ContainerStarted","Data":"3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760"} Oct 30 07:15:23.377534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.377508 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:23.683780 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:23.683739 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281e7884_05ef_49c5_a1a1_33a3bf42eda1.slice/crio-562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a WatchSource:0}: Error finding container 562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a: Status 404 returned error can't find the container with id 562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a Oct 30 07:15:23.684498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:23.684472 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697"] Oct 30 07:15:24.383230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.383196 5894 generic.go:356] "Generic (PLEG): container finished" podID="82661356-13e8-4a6e-ae5e-78b399bfd05a" containerID="40ebf7ba32180c68d01626d0d19d9fc5a03f9cfdfc376a9e7cc941e3cf5f1802" exitCode=0 Oct 30 07:15:24.383387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.383278 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" event={"ID":"82661356-13e8-4a6e-ae5e-78b399bfd05a","Type":"ContainerDied","Data":"40ebf7ba32180c68d01626d0d19d9fc5a03f9cfdfc376a9e7cc941e3cf5f1802"} Oct 30 07:15:24.385127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.385102 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" event={"ID":"281e7884-05ef-49c5-a1a1-33a3bf42eda1","Type":"ContainerStarted","Data":"3a4128d5e7d84182363e5bc71a80fbe0df2e1d686d24e0c92ed09de009f157b5"} Oct 30 07:15:24.385293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.385159 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" event={"ID":"281e7884-05ef-49c5-a1a1-33a3bf42eda1","Type":"ContainerStarted","Data":"562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a"} Oct 30 07:15:24.474179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.474116 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" podStartSLOduration=1.474097805 podStartE2EDuration="1.474097805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:24.471484069 +0000 UTC m=+1088.162401507" watchObservedRunningTime="2026-10-30 07:15:24.474097805 +0000 UTC m=+1088.165015244" Oct 30 07:15:24.710380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:24.710294 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:25.393209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.393173 5894 generic.go:356] "Generic (PLEG): container finished" podID="281e7884-05ef-49c5-a1a1-33a3bf42eda1" containerID="3a4128d5e7d84182363e5bc71a80fbe0df2e1d686d24e0c92ed09de009f157b5" exitCode=0 Oct 30 07:15:25.393426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.393256 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" event={"ID":"281e7884-05ef-49c5-a1a1-33a3bf42eda1","Type":"ContainerDied","Data":"3a4128d5e7d84182363e5bc71a80fbe0df2e1d686d24e0c92ed09de009f157b5"} Oct 30 07:15:25.578208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.578178 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:25.639686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.639650 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\" (UniqueName: \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\") pod \"82661356-13e8-4a6e-ae5e-78b399bfd05a\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " Oct 30 07:15:25.640074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.640047 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\") pod \"82661356-13e8-4a6e-ae5e-78b399bfd05a\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " Oct 30 07:15:25.640194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.640108 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\") pod \"82661356-13e8-4a6e-ae5e-78b399bfd05a\" (UID: \"82661356-13e8-4a6e-ae5e-78b399bfd05a\") " Oct 30 07:15:25.640501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.640439 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca" pod "82661356-13e8-4a6e-ae5e-78b399bfd05a" (UID: "82661356-13e8-4a6e-ae5e-78b399bfd05a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:25.640717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.640698 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:25.641917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.641892 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz" pod "82661356-13e8-4a6e-ae5e-78b399bfd05a" (UID: "82661356-13e8-4a6e-ae5e-78b399bfd05a"). InnerVolumeSpecName "kube-api-access-s5jqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:25.642283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.642256 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca" pod "82661356-13e8-4a6e-ae5e-78b399bfd05a" (UID: "82661356-13e8-4a6e-ae5e-78b399bfd05a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:25.741971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.741932 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jqz\" (UniqueName: \"kubernetes.io/projected/82661356-13e8-4a6e-ae5e-78b399bfd05a-kube-api-access-s5jqz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:25.741971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:25.741964 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82661356-13e8-4a6e-ae5e-78b399bfd05a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:26.269203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.269159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:26.382013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.381970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:26.401822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.401775 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" event={"ID":"82661356-13e8-4a6e-ae5e-78b399bfd05a","Type":"ContainerDied","Data":"3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760"} Oct 30 07:15:26.402017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.401843 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9021bf63f6306b9c36e0defef8e61c00226536c5d6fd051fdcef891ff03760" Oct 30 07:15:26.402017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.401797 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9" Oct 30 07:15:26.558335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.558310 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:26.654519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.654469 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\") pod \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " Oct 30 07:15:26.654724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.654536 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\") pod \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " Oct 30 07:15:26.654724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.654565 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\" (UniqueName: \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\") pod \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\" (UID: \"281e7884-05ef-49c5-a1a1-33a3bf42eda1\") " Oct 30 07:15:26.654974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.654949 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca" pod "281e7884-05ef-49c5-a1a1-33a3bf42eda1" (UID: "281e7884-05ef-49c5-a1a1-33a3bf42eda1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:26.655296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.655275 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:26.656742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.656718 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca" pod "281e7884-05ef-49c5-a1a1-33a3bf42eda1" (UID: "281e7884-05ef-49c5-a1a1-33a3bf42eda1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:26.656742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.656730 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm" pod "281e7884-05ef-49c5-a1a1-33a3bf42eda1" (UID: "281e7884-05ef-49c5-a1a1-33a3bf42eda1"). InnerVolumeSpecName "kube-api-access-hh7mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:26.756425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.756347 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh7mm\" (UniqueName: \"kubernetes.io/projected/281e7884-05ef-49c5-a1a1-33a3bf42eda1-kube-api-access-hh7mm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:26.756425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:26.756378 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/281e7884-05ef-49c5-a1a1-33a3bf42eda1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:27.410666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.410602 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" Oct 30 07:15:27.410831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.410677 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697" event={"ID":"281e7884-05ef-49c5-a1a1-33a3bf42eda1","Type":"ContainerDied","Data":"562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a"} Oct 30 07:15:27.410831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.410733 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="562bc0e92f9627ca4b7b403d07ea7236b288cd595f49d9a81bf1e1a95eb7345a" Oct 30 07:15:27.498028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.497996 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b"] Oct 30 07:15:27.498572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498559 5894 cpu_manager.go:414] "containerMap: removing container" podUID="281e7884-05ef-49c5-a1a1-33a3bf42eda1" containerName="revoke-keys" Oct 30 07:15:27.498624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498573 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281e7884-05ef-49c5-a1a1-33a3bf42eda1" containerName="revoke-keys" Oct 30 07:15:27.498624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498587 5894 cpu_manager.go:414] "containerMap: removing container" podUID="82661356-13e8-4a6e-ae5e-78b399bfd05a" containerName="revoke-keys" Oct 30 07:15:27.498624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498592 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82661356-13e8-4a6e-ae5e-78b399bfd05a" containerName="revoke-keys" Oct 30 07:15:27.498747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498728 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="281e7884-05ef-49c5-a1a1-33a3bf42eda1" containerName="revoke-keys" Oct 30 07:15:27.498747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.498739 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="82661356-13e8-4a6e-ae5e-78b399bfd05a" containerName="revoke-keys" Oct 30 07:15:27.502411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.502389 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.519490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.519445 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:15:27.519679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.519608 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b"] Oct 30 07:15:27.519794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.519771 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:15:27.529566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.529531 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:15:27.529711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.529566 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:15:27.567013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.566974 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9btv\" (UniqueName: \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.567198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.567093 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.567317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.567296 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.668462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.668378 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.668613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.668482 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.668613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.668558 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9btv\" (UniqueName: \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.669230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.669209 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.670774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.670753 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.694614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.694588 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9btv\" (UniqueName: \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\") pod \"maas-api-revoke-sub-4958a592-d8v9b\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:27.816249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.816200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:27.820067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:27.820045 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:28.143569 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:28.143528 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdc39e2_6892_491b_96d2_353a0f8ef512.slice/crio-3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd WatchSource:0}: Error finding container 3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd: Status 404 returned error can't find the container with id 3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd Oct 30 07:15:28.143983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:28.143962 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b"] Oct 30 07:15:28.420028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:28.419948 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" event={"ID":"0bdc39e2-6892-491b-96d2-353a0f8ef512","Type":"ContainerStarted","Data":"3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd"} Oct 30 07:15:29.430660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:29.429128 5894 generic.go:356] "Generic (PLEG): container finished" podID="0bdc39e2-6892-491b-96d2-353a0f8ef512" containerID="4deea02f20f062e5ce313470ad8070cae474e5ae8c9c0d95693c553c96d36878" exitCode=0 Oct 30 07:15:29.430660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:29.429295 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" event={"ID":"0bdc39e2-6892-491b-96d2-353a0f8ef512","Type":"ContainerDied","Data":"4deea02f20f062e5ce313470ad8070cae474e5ae8c9c0d95693c553c96d36878"} Oct 30 07:15:29.446697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:29.440627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:30.003359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.003324 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:30.033607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.033578 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv"] Oct 30 07:15:30.047664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.047603 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-862e9d30-rpdjv"] Oct 30 07:15:30.442441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.442402 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="610e6c9e-be60-4e46-a64f-d281a6ee59b1" path="/var/lib/kubelet/pods/610e6c9e-be60-4e46-a64f-d281a6ee59b1/volumes" Oct 30 07:15:30.483898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.483862 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:30.673654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.661847 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:30.673654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.664303 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:30.732719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.729181 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\" (UniqueName: \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\") pod \"0bdc39e2-6892-491b-96d2-353a0f8ef512\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " Oct 30 07:15:30.732719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.729248 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\") pod \"0bdc39e2-6892-491b-96d2-353a0f8ef512\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " Oct 30 07:15:30.732719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.729351 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\") pod \"0bdc39e2-6892-491b-96d2-353a0f8ef512\" (UID: \"0bdc39e2-6892-491b-96d2-353a0f8ef512\") " Oct 30 07:15:30.732719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.731008 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca" pod "0bdc39e2-6892-491b-96d2-353a0f8ef512" (UID: "0bdc39e2-6892-491b-96d2-353a0f8ef512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:15:30.749659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.743914 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv" pod "0bdc39e2-6892-491b-96d2-353a0f8ef512" (UID: "0bdc39e2-6892-491b-96d2-353a0f8ef512"). InnerVolumeSpecName "kube-api-access-x9btv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:15:30.770891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.767512 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca" pod "0bdc39e2-6892-491b-96d2-353a0f8ef512" (UID: "0bdc39e2-6892-491b-96d2-353a0f8ef512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:15:30.831919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.831871 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:30.831919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.831911 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9btv\" (UniqueName: \"kubernetes.io/projected/0bdc39e2-6892-491b-96d2-353a0f8ef512-kube-api-access-x9btv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:30.832124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:30.831930 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bdc39e2-6892-491b-96d2-353a0f8ef512-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:15:31.019977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:31.019883 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:31.454812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:31.454779 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" Oct 30 07:15:31.454812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:31.454796 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b" event={"ID":"0bdc39e2-6892-491b-96d2-353a0f8ef512","Type":"ContainerDied","Data":"3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd"} Oct 30 07:15:31.455294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:31.454857 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3af7e980744e0231cbe9415c241faeb6d0535df188cfceef7ecef0585ba62ffd" Oct 30 07:15:32.708360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:32.708312 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:33.454600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:33.454560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.220267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.220233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.337829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.337791 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.472534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.472439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.611167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.611133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.776098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.775983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:34.829767 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:15:34.829731 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:15:34.928468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:34.925195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:36.068187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:36.068149 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:36.200401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:36.200369 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:37.792183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:37.792152 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:38.521337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:38.521296 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:38.715431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:38.715397 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:39.546078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:39.546035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:40.217683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:40.211894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:40.391101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:40.391072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:41.562720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:41.562681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:42.548170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:42.548133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:42.931261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:42.931166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:43.033473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.033439 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm"] Oct 30 07:15:43.049664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.049592 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4ab7a28-mcrkm"] Oct 30 07:15:43.565923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.565843 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:43.687539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.687461 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:15:43.688406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.688381 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0bdc39e2-6892-491b-96d2-353a0f8ef512" containerName="revoke-keys" Oct 30 07:15:43.688406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.688402 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bdc39e2-6892-491b-96d2-353a0f8ef512" containerName="revoke-keys" Oct 30 07:15:43.688586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.688542 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bdc39e2-6892-491b-96d2-353a0f8ef512" containerName="revoke-keys" Oct 30 07:15:43.691716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.691683 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:43.692735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.692715 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733168 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733225 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733255 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733310 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rbv\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733471 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733597 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733734 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.733950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.733810 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.734300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.734255 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.735208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.735183 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:15:43.739771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.739748 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:15:43.744336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.744313 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.803050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.803009 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:15:43.826518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.826421 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:43.835250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835218 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835296 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835323 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.835386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835355 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.835524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835575 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835600 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rbv\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835624 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.835696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835679 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.835878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835763 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.835878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835806 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.835878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835844 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.836068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.835946 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.836068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836050 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.836173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836089 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.836173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836157 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxbf\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.836284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836190 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.836284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836220 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.836284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836232 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.836284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.836244 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.837035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.837004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.837147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.837037 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.838053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.838022 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.838991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.838955 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.839275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.839256 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.861686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.861295 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.861686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.861417 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rbv\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:43.938037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.937994 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxbf\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938038 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938054 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938203 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938239 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938275 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938313 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938367 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938402 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.938766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938580 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.939113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938474 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.939113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.938910 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.939226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.939127 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.939226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.939173 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.940683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.940661 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.941090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.941067 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.963516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.963484 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:43.963680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:43.963619 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxbf\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:44.008963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.008923 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:44.099408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.099324 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:44.330734 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:44.330707 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16ed888_fd3a_4219_8440_d1ee0dc6e82a.slice/crio-660da3d4e1885838cb4d46bbea84a7018f5f2f884b12be6643ab0aeebff89798 WatchSource:0}: Error finding container 660da3d4e1885838cb4d46bbea84a7018f5f2f884b12be6643ab0aeebff89798: Status 404 returned error can't find the container with id 660da3d4e1885838cb4d46bbea84a7018f5f2f884b12be6643ab0aeebff89798 Oct 30 07:15:44.332264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.332235 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:15:44.333354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.333325 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.333440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.333387 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.333440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.333419 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.434829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.434801 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3" path="/var/lib/kubelet/pods/3b5fac62-6f0d-4b2d-bb81-f1ed38559cd3/volumes" Oct 30 07:15:44.436947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.436920 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:15:44.437148 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:15:44.437126 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be0c87d_1f01_4978_af59_f863085abf27.slice/crio-99de004d6532634557372fd33a289374750ce95a7e2174106a636f38413e281b WatchSource:0}: Error finding container 99de004d6532634557372fd33a289374750ce95a7e2174106a636f38413e281b: Status 404 returned error can't find the container with id 99de004d6532634557372fd33a289374750ce95a7e2174106a636f38413e281b Oct 30 07:15:44.439611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.439583 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.439747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.439682 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.439747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.439726 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:15:44.569595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.569555 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" event={"ID":"3be0c87d-1f01-4978-af59-f863085abf27","Type":"ContainerStarted","Data":"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b"} Oct 30 07:15:44.569818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.569623 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" event={"ID":"3be0c87d-1f01-4978-af59-f863085abf27","Type":"ContainerStarted","Data":"99de004d6532634557372fd33a289374750ce95a7e2174106a636f38413e281b"} Oct 30 07:15:44.571804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.571771 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" event={"ID":"e16ed888-fd3a-4219-8440-d1ee0dc6e82a","Type":"ContainerStarted","Data":"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57"} Oct 30 07:15:44.571941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.571830 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" event={"ID":"e16ed888-fd3a-4219-8440-d1ee0dc6e82a","Type":"ContainerStarted","Data":"660da3d4e1885838cb4d46bbea84a7018f5f2f884b12be6643ab0aeebff89798"} Oct 30 07:15:44.630589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.630476 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podStartSLOduration=1.630456908 podStartE2EDuration="1.630456908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:44.624099776 +0000 UTC m=+1108.315017214" watchObservedRunningTime="2026-10-30 07:15:44.630456908 +0000 UTC m=+1108.321374349" Oct 30 07:15:44.678322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:44.678266 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podStartSLOduration=1.678248774 podStartE2EDuration="1.678248774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:15:44.674332031 +0000 UTC m=+1108.365249469" watchObservedRunningTime="2026-10-30 07:15:44.678248774 +0000 UTC m=+1108.369166210" Oct 30 07:15:45.009681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.009559 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:45.011390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.011364 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:45.011528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.011412 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:45.034117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.034086 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n"] Oct 30 07:15:45.037942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.037912 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9"] Oct 30 07:15:45.043099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.043061 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h"] Oct 30 07:15:45.047370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.047336 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba63d382-zkv2n"] Oct 30 07:15:45.050894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.050872 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5f9fc42-mwv9h"] Oct 30 07:15:45.054201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.054176 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d187f494-s5jp9"] Oct 30 07:15:45.100249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.100217 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:45.101723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.101698 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:45.101806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.101747 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:45.567259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.567227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:45.700656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.700108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:45.944866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:45.944770 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:46.016655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.016109 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:46.016655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.016172 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:46.100659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.100111 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:46.100659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.100170 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:46.469521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.469390 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a5f763-fdce-430e-bb11-32f3e34fdc97" path="/var/lib/kubelet/pods/04a5f763-fdce-430e-bb11-32f3e34fdc97/volumes" Oct 30 07:15:46.474687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.470815 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dad459d-9254-42a0-8094-3c1f29a98553" path="/var/lib/kubelet/pods/3dad459d-9254-42a0-8094-3c1f29a98553/volumes" Oct 30 07:15:46.474687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:46.471556 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8d6a99-6390-4f97-b4fd-44a2cf84cce7" path="/var/lib/kubelet/pods/4b8d6a99-6390-4f97-b4fd-44a2cf84cce7/volumes" Oct 30 07:15:47.012661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.012165 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:47.012661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.012222 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:47.117659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.110169 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:47.117659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.110230 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:47.528331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.528291 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:47.886720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:47.882614 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:48.011656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:48.011039 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:48.011656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:48.011113 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:48.113662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:48.109054 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:48.113662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:48.109115 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:49.012159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:49.012020 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:49.012159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:49.012098 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:49.104126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:49.104084 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:49.104535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:49.104501 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:49.391658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:49.390964 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:50.010789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:50.010749 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:50.011093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:50.011060 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:50.104660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:50.102018 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:50.104660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:50.102080 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:51.009665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.009595 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 07:15:51.009859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.009718 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 07:15:51.109474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.109437 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 07:15:51.110027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.109503 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 07:15:51.330065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.330027 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:51.458142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:51.458109 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:52.158539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.155544 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:52.159820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.159746 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:52.162218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.162079 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:15:52.694315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.694281 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:52.694786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.694763 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:52.695920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:52.695898 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:15:53.130955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:53.130897 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:54.719405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:54.719368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:55.339321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:55.339284 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:55.456523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:55.456492 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:56.094407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:56.094350 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:56.256946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:56.256919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:56.456982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:56.456821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:58.200175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:58.200140 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:58.338701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:58.338665 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:58.470021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:58.469929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:15:59.998522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:15:59.998489 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:01.526854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:01.526815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:03.035935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:03.035903 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s"] Oct 30 07:16:03.048113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:03.048074 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca32fce9-gcx5s"] Oct 30 07:16:03.175288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:03.175254 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:04.434732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:04.434687 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d214a166-1290-481a-b587-fb9a13208115" path="/var/lib/kubelet/pods/d214a166-1290-481a-b587-fb9a13208115/volumes" Oct 30 07:16:04.709928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:04.709831 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:04.820281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:04.820239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:04.837273 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:04.837208 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:16:05.391804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:05.391771 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:05.871435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:05.871394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:06.216087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:06.215989 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:06.469072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:06.468987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:08.115341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:08.115310 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:08.833773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:08.833732 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:09.137694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:09.137579 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:16:09.578797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:09.578753 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:10.026179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:10.026095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:10.190879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:10.190844 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:10.388456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:10.388423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:12.137206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:12.137172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:12.300202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:12.300166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:12.520716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:12.520390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:13.039029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:13.038978 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs"] Oct 30 07:16:13.055918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:13.055861 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65f14b52-q2lgs"] Oct 30 07:16:14.114154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:14.114117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:14.432165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:14.432086 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f3bcb5-9d95-4804-aac3-c703a52fb50e" path="/var/lib/kubelet/pods/96f3bcb5-9d95-4804-aac3-c703a52fb50e/volumes" Oct 30 07:16:14.471708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:14.471675 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:14.601783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:14.601749 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:16.233820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:16.233695 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:16.728680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:16.728594 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:16.740412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:16.740364 5894 scope.go:122] "RemoveContainer" containerID="5fe7dfa342e35a453c97d2b4ef017c9d4edcfd3e59e0007a989fe8332fb0ce98" Oct 30 07:16:16.851578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:16.851544 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:18.046199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:18.046170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:19.712181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.712146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:19.780572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.780531 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq"] Oct 30 07:16:19.789575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.789527 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.802702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.801860 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq"] Oct 30 07:16:19.809185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.808963 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:16:19.809185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.808982 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:16:19.809185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.809014 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:16:19.809185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.809145 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:16:19.851961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.851921 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.852286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.852254 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.852442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.852332 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7j2\" (UniqueName: \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.953439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.953399 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.953683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.953559 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.953683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.953614 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7j2\" (UniqueName: \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.954223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.954197 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.955821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.955798 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:19.978784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:19.978723 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7j2\" (UniqueName: \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\") pod \"maas-api-revoke-sub-ac1b07b6-28vgq\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:20.131452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:20.131419 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:20.439877 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:20.439836 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f80afe2_da7f_4d2f_85dc_1900abd04a1e.slice/crio-0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc WatchSource:0}: Error finding container 0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc: Status 404 returned error can't find the container with id 0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc Oct 30 07:16:20.440005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:20.439983 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq"] Oct 30 07:16:21.087118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:21.087080 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" event={"ID":"7f80afe2-da7f-4d2f-85dc-1900abd04a1e","Type":"ContainerStarted","Data":"33ebcf2ef9b86be9eb79c9365ff5dc7bb3bda56de260ad35473ead1286695ab8"} Oct 30 07:16:21.087528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:21.087163 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" event={"ID":"7f80afe2-da7f-4d2f-85dc-1900abd04a1e","Type":"ContainerStarted","Data":"0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc"} Oct 30 07:16:21.132790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:21.132737 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" podStartSLOduration=2.1327228480000002 podStartE2EDuration="2.132722848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:21.131668862 +0000 UTC m=+1144.822586301" watchObservedRunningTime="2026-10-30 07:16:21.132722848 +0000 UTC m=+1144.823640293" Oct 30 07:16:21.207918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:21.207856 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:21.472280 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:21.472245 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f80afe2_da7f_4d2f_85dc_1900abd04a1e.slice/crio-33ebcf2ef9b86be9eb79c9365ff5dc7bb3bda56de260ad35473ead1286695ab8.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:16:22.097014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:22.096970 5894 generic.go:356] "Generic (PLEG): container finished" podID="7f80afe2-da7f-4d2f-85dc-1900abd04a1e" containerID="33ebcf2ef9b86be9eb79c9365ff5dc7bb3bda56de260ad35473ead1286695ab8" exitCode=0 Oct 30 07:16:22.097412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:22.097053 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" event={"ID":"7f80afe2-da7f-4d2f-85dc-1900abd04a1e","Type":"ContainerDied","Data":"33ebcf2ef9b86be9eb79c9365ff5dc7bb3bda56de260ad35473ead1286695ab8"} Oct 30 07:16:22.706297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:22.706254 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:23.266967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.266939 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:23.310333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.310303 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7605-0c6144"] Oct 30 07:16:23.310929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.310915 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7f80afe2-da7f-4d2f-85dc-1900abd04a1e" containerName="revoke-keys" Oct 30 07:16:23.310985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.310931 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f80afe2-da7f-4d2f-85dc-1900abd04a1e" containerName="revoke-keys" Oct 30 07:16:23.311075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.311066 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f80afe2-da7f-4d2f-85dc-1900abd04a1e" containerName="revoke-keys" Oct 30 07:16:23.315105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.315078 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:23.332148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.332117 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-0c6144"] Oct 30 07:16:23.335274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.335250 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:16:23.402356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.402274 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\") pod \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " Oct 30 07:16:23.402512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.402482 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\") pod \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " Oct 30 07:16:23.402595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.402579 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\" (UniqueName: \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\") pod \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\" (UID: \"7f80afe2-da7f-4d2f-85dc-1900abd04a1e\") " Oct 30 07:16:23.402667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.402611 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca" pod "7f80afe2-da7f-4d2f-85dc-1900abd04a1e" (UID: "7f80afe2-da7f-4d2f-85dc-1900abd04a1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:23.402986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.402966 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkfnf\" (UniqueName: \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\") pod \"test-curl-7605-0c6144\" (UID: \"55fc5f33-e2b0-4b4d-8413-99c1954036ee\") " pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:23.403288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.403263 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:23.404751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.404722 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2" pod "7f80afe2-da7f-4d2f-85dc-1900abd04a1e" (UID: "7f80afe2-da7f-4d2f-85dc-1900abd04a1e"). InnerVolumeSpecName "kube-api-access-8k7j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:23.404862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.404806 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca" pod "7f80afe2-da7f-4d2f-85dc-1900abd04a1e" (UID: "7f80afe2-da7f-4d2f-85dc-1900abd04a1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:16:23.504272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.504214 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkfnf\" (UniqueName: \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\") pod \"test-curl-7605-0c6144\" (UID: \"55fc5f33-e2b0-4b4d-8413-99c1954036ee\") " pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:23.504964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.504941 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:23.504964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.504963 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k7j2\" (UniqueName: \"kubernetes.io/projected/7f80afe2-da7f-4d2f-85dc-1900abd04a1e-kube-api-access-8k7j2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:23.529551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.529518 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkfnf\" (UniqueName: \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\") pod \"test-curl-7605-0c6144\" (UID: \"55fc5f33-e2b0-4b4d-8413-99c1954036ee\") " pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:23.634866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.634834 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:23.951237 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:23.951194 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55fc5f33_e2b0_4b4d_8413_99c1954036ee.slice/crio-0b791ce5e79696e72228d66d4e99262fa131fb501d080454ce8f95263e395f6f WatchSource:0}: Error finding container 0b791ce5e79696e72228d66d4e99262fa131fb501d080454ce8f95263e395f6f: Status 404 returned error can't find the container with id 0b791ce5e79696e72228d66d4e99262fa131fb501d080454ce8f95263e395f6f Oct 30 07:16:23.953316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:23.953292 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7605-0c6144"] Oct 30 07:16:24.112400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:24.112368 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-0c6144" event={"ID":"55fc5f33-e2b0-4b4d-8413-99c1954036ee","Type":"ContainerStarted","Data":"0b791ce5e79696e72228d66d4e99262fa131fb501d080454ce8f95263e395f6f"} Oct 30 07:16:24.113799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:24.113777 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" event={"ID":"7f80afe2-da7f-4d2f-85dc-1900abd04a1e","Type":"ContainerDied","Data":"0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc"} Oct 30 07:16:24.113928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:24.113809 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq" Oct 30 07:16:24.113928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:24.113815 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e5551426beb9a6bafc0d0a234dd5b0417b007799802756b0acc3659cf35e2bc" Oct 30 07:16:24.358366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:24.358327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:25.125322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:25.125284 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7605-0c6144" event={"ID":"55fc5f33-e2b0-4b4d-8413-99c1954036ee","Type":"ContainerStarted","Data":"7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224"} Oct 30 07:16:25.169227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:25.169171 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7605-0c6144" podStartSLOduration=1.539674816 podStartE2EDuration="2.169156649s" totalImagesPullingTime="629.481833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:25.166725702 +0000 UTC m=+1148.857643141" watchObservedRunningTime="2026-10-30 07:16:25.169156649 +0000 UTC m=+1148.860074077" Oct 30 07:16:25.887013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:25.886972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:26.001397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.001346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:26.099824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.099785 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7605-0c6144"] Oct 30 07:16:26.106698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.105182 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7605-0c6144"] Oct 30 07:16:26.118865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.118819 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:26.379443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.379412 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv"] Oct 30 07:16:26.380085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.380064 5894 cpu_manager.go:414] "containerMap: removing container" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" containerName="test-curl-7605-0c6144" Oct 30 07:16:26.380085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.380083 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" containerName="test-curl-7605-0c6144" Oct 30 07:16:26.380248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.380207 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" containerName="test-curl-7605-0c6144" Oct 30 07:16:26.384094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.384077 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.415693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.415660 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:16:26.415693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.415692 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:16:26.422013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.421974 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv"] Oct 30 07:16:26.430009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.429958 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:16:26.430213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.430197 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:16:26.450873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.450844 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.451065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.451037 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq69m\" (UniqueName: \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.451130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.451091 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.497400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.497368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:26.552340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.552293 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.552551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.552532 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq69m\" (UniqueName: \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.552626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.552609 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.553417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.553388 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.554871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.554838 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.578298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.578265 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq69m\" (UniqueName: \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\") pod \"maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:26.698563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:26.698475 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:27.028764 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:27.028732 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd57e0c32_be52_4043_b3f7_c72671370fc0.slice/crio-94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d WatchSource:0}: Error finding container 94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d: Status 404 returned error can't find the container with id 94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d Oct 30 07:16:27.029134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:27.029077 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv"] Oct 30 07:16:27.146925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:27.146878 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" event={"ID":"d57e0c32-be52-4043-b3f7-c72671370fc0","Type":"ContainerStarted","Data":"94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d"} Oct 30 07:16:27.147126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:27.146895 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7605-0c6144" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" containerName="test-curl-7605-0c6144" containerID="cri-o://7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224" gracePeriod=2 Oct 30 07:16:27.699466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:27.699365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:28.156991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:28.156950 5894 generic.go:356] "Generic (PLEG): container finished" podID="d57e0c32-be52-4043-b3f7-c72671370fc0" containerID="a0a57f8f4ca6e6ff13ab1bb349eefec15c8a93dc5009f0b0b386877dc3c75e96" exitCode=0 Oct 30 07:16:28.157480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:28.157236 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" event={"ID":"d57e0c32-be52-4043-b3f7-c72671370fc0","Type":"ContainerDied","Data":"a0a57f8f4ca6e6ff13ab1bb349eefec15c8a93dc5009f0b0b386877dc3c75e96"} Oct 30 07:16:29.192780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.189984 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:29.380452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.380400 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:29.393688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.393658 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:29.499108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499021 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\") pod \"d57e0c32-be52-4043-b3f7-c72671370fc0\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " Oct 30 07:16:29.499108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499083 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\" (UniqueName: \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\") pod \"d57e0c32-be52-4043-b3f7-c72671370fc0\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " Oct 30 07:16:29.499297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499124 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\") pod \"d57e0c32-be52-4043-b3f7-c72671370fc0\" (UID: \"d57e0c32-be52-4043-b3f7-c72671370fc0\") " Oct 30 07:16:29.499297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499227 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\" (UniqueName: \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\") pod \"55fc5f33-e2b0-4b4d-8413-99c1954036ee\" (UID: \"55fc5f33-e2b0-4b4d-8413-99c1954036ee\") " Oct 30 07:16:29.499550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499523 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca" pod "d57e0c32-be52-4043-b3f7-c72671370fc0" (UID: "d57e0c32-be52-4043-b3f7-c72671370fc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:29.499865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.499821 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:29.501262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.501236 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca" pod "d57e0c32-be52-4043-b3f7-c72671370fc0" (UID: "d57e0c32-be52-4043-b3f7-c72671370fc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:16:29.501368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.501337 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m" pod "d57e0c32-be52-4043-b3f7-c72671370fc0" (UID: "d57e0c32-be52-4043-b3f7-c72671370fc0"). InnerVolumeSpecName "kube-api-access-wq69m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:29.501368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.501359 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf" pod "55fc5f33-e2b0-4b4d-8413-99c1954036ee" (UID: "55fc5f33-e2b0-4b4d-8413-99c1954036ee"). InnerVolumeSpecName "kube-api-access-mkfnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:29.600925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.600885 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkfnf\" (UniqueName: \"kubernetes.io/projected/55fc5f33-e2b0-4b4d-8413-99c1954036ee-kube-api-access-mkfnf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:29.600925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.600920 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d57e0c32-be52-4043-b3f7-c72671370fc0-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:29.600925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:29.600931 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq69m\" (UniqueName: \"kubernetes.io/projected/d57e0c32-be52-4043-b3f7-c72671370fc0-kube-api-access-wq69m\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:30.174183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.174147 5894 generic.go:356] "Generic (PLEG): container finished" podID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" containerID="7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224" exitCode=137 Oct 30 07:16:30.174381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.174194 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:30.174381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.174249 5894 scope.go:122] "RemoveContainer" containerID="7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224" Oct 30 07:16:30.176200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.176175 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" event={"ID":"d57e0c32-be52-4043-b3f7-c72671370fc0","Type":"ContainerDied","Data":"94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d"} Oct 30 07:16:30.176331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.176257 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94edff4d83ffa1e8aa86615cf0043d9270288a004da4b0c2120c91ca7e10161d" Oct 30 07:16:30.176398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.176322 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv" Oct 30 07:16:30.187288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.187263 5894 scope.go:122] "RemoveContainer" containerID="7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224" Oct 30 07:16:30.187686 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:30.187619 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224\": container with ID starting with 7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224 not found: ID does not exist" containerID="7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224" Oct 30 07:16:30.187774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.187692 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224"} err="failed to get container status \"7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224\": rpc error: code = NotFound desc = could not find container \"7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224\": container with ID starting with 7e59c09dcdde8c08c70c1186fae53b0519240cbfb77990f72f423d006289a224 not found: ID does not exist" Oct 30 07:16:30.221644 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:30.221608 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-0c6144\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:30.237297 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:30.237261 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-0c6144\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:30.395511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.395472 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:16:30.396149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.396134 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d57e0c32-be52-4043-b3f7-c72671370fc0" containerName="revoke-keys" Oct 30 07:16:30.396233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.396159 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d57e0c32-be52-4043-b3f7-c72671370fc0" containerName="revoke-keys" Oct 30 07:16:30.396270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.396251 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d57e0c32-be52-4043-b3f7-c72671370fc0" containerName="revoke-keys" Oct 30 07:16:30.401324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.401302 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.417775 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:30.417743 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7605-0c6144\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" pod="opendatahub/test-curl-7605-0c6144" Oct 30 07:16:30.419910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.419886 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d0dace8c-openshift-default-dockercfg-lknns\"" Oct 30 07:16:30.435622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.435527 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55fc5f33-e2b0-4b4d-8413-99c1954036ee" path="/var/lib/kubelet/pods/55fc5f33-e2b0-4b4d-8413-99c1954036ee/volumes" Oct 30 07:16:30.436062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.436036 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:16:30.487084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.487053 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:30.517237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517198 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517237 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517473 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517563 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517587 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517629 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517797 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm8xt\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.517888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.517869 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.518137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.518112 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619149 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619185 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619321 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619398 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm8xt\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619443 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619504 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619560 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619576 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619648 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619688 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619624 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.619967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619945 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.620032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.619986 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.620270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.620244 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.621903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.621879 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.621990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.621943 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.663984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.663938 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm8xt\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.664188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.664162 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\") pod \"e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.715787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.715673 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:30.732808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.732778 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:30.859169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:30.859126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:31.028191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.026347 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:31.051518 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:31.051479 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75992f34_ee68_4ed4_bcf3_9d5209a52097.slice/crio-557da1e6d5814c4915759132c87de8405dd662fcfd6a201e8b740bd0335740e7 WatchSource:0}: Error finding container 557da1e6d5814c4915759132c87de8405dd662fcfd6a201e8b740bd0335740e7: Status 404 returned error can't find the container with id 557da1e6d5814c4915759132c87de8405dd662fcfd6a201e8b740bd0335740e7 Oct 30 07:16:31.053101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.053071 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:16:31.054408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.054372 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:31.054498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.054448 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:31.054577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.054494 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:31.223688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.212491 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:31.242861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:31.232373 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" event={"ID":"75992f34-ee68-4ed4-bcf3-9d5209a52097","Type":"ContainerStarted","Data":"557da1e6d5814c4915759132c87de8405dd662fcfd6a201e8b740bd0335740e7"} Oct 30 07:16:32.249724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.248191 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" event={"ID":"75992f34-ee68-4ed4-bcf3-9d5209a52097","Type":"ContainerStarted","Data":"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662"} Oct 30 07:16:32.344139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.344065 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podStartSLOduration=2.344041385 podStartE2EDuration="2.344041385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:32.324609026 +0000 UTC m=+1156.015526464" watchObservedRunningTime="2026-10-30 07:16:32.344041385 +0000 UTC m=+1156.034958824" Oct 30 07:16:32.542604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.542564 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:32.718273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.718139 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:32.719223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.719191 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:32.719373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:32.719249 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:33.716078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:33.716047 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:33.716469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:33.716100 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:34.235099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:34.235063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:34.716252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:34.716214 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:34.716672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:34.716273 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:34.829768 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:16:34.829726 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:16:35.336297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:35.336249 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:35.491579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:35.491543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:35.716257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:35.716175 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:35.716257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:35.716226 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:36.057228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:36.057186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:36.716901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:36.716865 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:36.717309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:36.716926 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:37.005600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.005509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:37.103348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.103309 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:16:37.108852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.108826 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.127557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.127517 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:37.132666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.132596 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:16:37.139602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.139573 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-318524\"" Oct 30 07:16:37.167538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.167499 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.167835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.167813 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7tz\" (UniqueName: \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.199722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.191250 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:16:37.207078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.205953 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:16:37.207078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.206053 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.223324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.223290 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-318524-dockercfg-qfgcg\"" Oct 30 07:16:37.442904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.442854 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.443093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.442916 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7tz\" (UniqueName: \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.443093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.443058 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.443249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.443097 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqk2g\" (UniqueName: \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.444659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.444561 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.469185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.469152 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7tz\" (UniqueName: \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\") pod \"payload-processing-e2e-api-a-318524-66755578d-8zlrz\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.544093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.544057 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqk2g\" (UniqueName: \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.544643 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.544595 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.546227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.546202 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.569251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.569221 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqk2g\" (UniqueName: \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\") pod \"payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.716322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.716231 5894 patch_prober.go:29] interesting pod/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 07:16:37.716494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.716318 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 07:16:37.733652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.733324 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:37.735926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.735898 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:37.857750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.837781 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:37.864271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:37.864005 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:38.051389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.051358 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:16:38.092193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.092164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:38.209908 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:38.209878 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb4e048_b97d_4b33_b9ad_0f5c6d147b66.slice/crio-fa2b3e16fa8685dc8a4cf16ee982ca70a6b4aedefc62f4cf0dcd97fe3d10a27b WatchSource:0}: Error finding container fa2b3e16fa8685dc8a4cf16ee982ca70a6b4aedefc62f4cf0dcd97fe3d10a27b: Status 404 returned error can't find the container with id fa2b3e16fa8685dc8a4cf16ee982ca70a6b4aedefc62f4cf0dcd97fe3d10a27b Oct 30 07:16:38.212468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.212439 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:16:38.462169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.462134 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" event={"ID":"860e0667-394a-4cc8-93cc-f2694b5a178d","Type":"ContainerStarted","Data":"56dc211f9282a361c95009bcfe6ba6c7e35fa415264e34c740da79aa1940b024"} Oct 30 07:16:38.463700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.463665 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" event={"ID":"abb4e048-b97d-4b33-b9ad-0f5c6d147b66","Type":"ContainerStarted","Data":"fa2b3e16fa8685dc8a4cf16ee982ca70a6b4aedefc62f4cf0dcd97fe3d10a27b"} Oct 30 07:16:38.989310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.989114 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:38.990360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.990265 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:38.991535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:38.991226 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:16:39.931673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:39.929802 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:16:39.958972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:39.958936 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:41.502200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.502158 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" event={"ID":"860e0667-394a-4cc8-93cc-f2694b5a178d","Type":"ContainerStarted","Data":"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03"} Oct 30 07:16:41.502769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.502270 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:41.504830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.504796 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" event={"ID":"abb4e048-b97d-4b33-b9ad-0f5c6d147b66","Type":"ContainerStarted","Data":"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766"} Oct 30 07:16:41.504956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.504855 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:41.558323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.558266 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" podStartSLOduration=1.886397243 podStartE2EDuration="4.558250344s" totalImagesPullingTime="2.671853101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:41.555335085 +0000 UTC m=+1165.246252523" watchObservedRunningTime="2026-10-30 07:16:41.558250344 +0000 UTC m=+1165.249167804" Oct 30 07:16:41.605870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.605796 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" podStartSLOduration=1.984528356 podStartE2EDuration="4.60577607s" totalImagesPullingTime="2.621247714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:41.602210674 +0000 UTC m=+1165.293128112" watchObservedRunningTime="2026-10-30 07:16:41.60577607 +0000 UTC m=+1165.296693508" Oct 30 07:16:41.935611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:41.935558 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:43.772126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:43.772080 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:45.318052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:45.318021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:47.030460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:47.030415 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:48.766504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:48.766472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:50.327990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:50.327943 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:50.631586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:50.631502 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:51.712358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:51.712321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:52.435449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.435396 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:52.537842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.537802 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv"] Oct 30 07:16:52.543591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.543560 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:16:52.543875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.543758 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:16:52.550134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.550103 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d0dace8c-m9jmv"] Oct 30 07:16:52.566375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.566340 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:52.715025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.714912 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:52.827820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.827779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:52.860863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.860828 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84"] Oct 30 07:16:52.869725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.869690 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:52.888695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.886171 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84"] Oct 30 07:16:52.888695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.887444 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:16:52.888695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.887541 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:16:52.966242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.966151 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:16:52.966242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:52.966187 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:16:53.020793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.020747 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.020986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.020873 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.020986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.020900 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrr6\" (UniqueName: \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.060725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.060691 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:53.122281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.122240 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.122485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.122319 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.122485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.122341 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrr6\" (UniqueName: \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.123087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.123066 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.124683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.124655 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.147992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.147960 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrr6\" (UniqueName: \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\") pod \"maas-api-revoke-keys-e2e-api-a-77ef52-s4s84\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.193836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.193797 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:53.507383 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:53.507347 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d598ee3_6ee2_4a08_bd1a_8b9285d94078.slice/crio-b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3 WatchSource:0}: Error finding container b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3: Status 404 returned error can't find the container with id b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3 Oct 30 07:16:53.510830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.510806 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84"] Oct 30 07:16:53.656780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:53.656740 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" event={"ID":"3d598ee3-6ee2-4a08-bd1a-8b9285d94078","Type":"ContainerStarted","Data":"b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3"} Oct 30 07:16:54.217608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:54.217516 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:54.433365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:54.433324 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d57e0c32-be52-4043-b3f7-c72671370fc0" path="/var/lib/kubelet/pods/d57e0c32-be52-4043-b3f7-c72671370fc0/volumes" Oct 30 07:16:54.665285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:54.665245 5894 generic.go:356] "Generic (PLEG): container finished" podID="3d598ee3-6ee2-4a08-bd1a-8b9285d94078" containerID="0048ff9d84671d0e1c38efe2935f4413e2fd38e1180ed4ee6fc9b1d64a5c771d" exitCode=0 Oct 30 07:16:54.665462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:54.665336 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" event={"ID":"3d598ee3-6ee2-4a08-bd1a-8b9285d94078","Type":"ContainerDied","Data":"0048ff9d84671d0e1c38efe2935f4413e2fd38e1180ed4ee6fc9b1d64a5c771d"} Oct 30 07:16:54.689937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:54.689901 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:55.297956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.297920 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:16:55.302951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.302920 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.349449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.349406 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:16:55.408495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.408453 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-56eb7f-openshift-default-dockercfg-qcrln\"" Oct 30 07:16:55.466742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.466702 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.466940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.466821 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.466940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.466908 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.466960 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.466985 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.467010 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.467228 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.467390 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvm2q\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.467497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.467483 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.568973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.568866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.568973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.568923 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.568973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.568962 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569098 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvm2q\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569190 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569267 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569306 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569350 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569377 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569539 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.569599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569585 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.570013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.569986 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.570123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.570023 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.570214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.570193 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.571499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.571480 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.571862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.571838 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.620745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.620709 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.621196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.621172 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvm2q\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.837807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.837778 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:55.874700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.874666 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\") pod \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " Oct 30 07:16:55.874865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.874844 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\" (UniqueName: \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\") pod \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " Oct 30 07:16:55.874944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.874927 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\") pod \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\" (UID: \"3d598ee3-6ee2-4a08-bd1a-8b9285d94078\") " Oct 30 07:16:55.875264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.875240 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca" pod "3d598ee3-6ee2-4a08-bd1a-8b9285d94078" (UID: "3d598ee3-6ee2-4a08-bd1a-8b9285d94078"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:55.875441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.875424 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:55.876880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.876861 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca" pod "3d598ee3-6ee2-4a08-bd1a-8b9285d94078" (UID: "3d598ee3-6ee2-4a08-bd1a-8b9285d94078"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:16:55.876951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.876905 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6" pod "3d598ee3-6ee2-4a08-bd1a-8b9285d94078" (UID: "3d598ee3-6ee2-4a08-bd1a-8b9285d94078"). InnerVolumeSpecName "kube-api-access-vvrr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:55.906674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.906613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:55.919277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.919245 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:55.983662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.978920 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:55.983662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:55.978952 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrr6\" (UniqueName: \"kubernetes.io/projected/3d598ee3-6ee2-4a08-bd1a-8b9285d94078-kube-api-access-vvrr6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:56.135713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.135568 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm"] Oct 30 07:16:56.136265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.136244 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3d598ee3-6ee2-4a08-bd1a-8b9285d94078" containerName="revoke-keys" Oct 30 07:16:56.136265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.136261 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d598ee3-6ee2-4a08-bd1a-8b9285d94078" containerName="revoke-keys" Oct 30 07:16:56.136450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.136356 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d598ee3-6ee2-4a08-bd1a-8b9285d94078" containerName="revoke-keys" Oct 30 07:16:56.141270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.141242 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.159497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.159466 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm"] Oct 30 07:16:56.274774 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:56.274739 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b2f4984_85ca_4da3_9d66_9de09c972a8c.slice/crio-ac9caa636378dceda6c76a00d64abf29a48eff4b9ea54b5c0609ed2023dfeef0 WatchSource:0}: Error finding container ac9caa636378dceda6c76a00d64abf29a48eff4b9ea54b5c0609ed2023dfeef0: Status 404 returned error can't find the container with id ac9caa636378dceda6c76a00d64abf29a48eff4b9ea54b5c0609ed2023dfeef0 Oct 30 07:16:56.276072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.276043 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:16:56.277472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.277438 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.277557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.277501 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.277557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.277530 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:16:56.283517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.283494 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.283696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.283674 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k26jx\" (UniqueName: \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.283765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.283719 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.385272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.385221 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.385865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.385436 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k26jx\" (UniqueName: \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.385865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.385488 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.386000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.385924 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.388002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.387974 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.411521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.411483 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k26jx\" (UniqueName: \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\") pod \"maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.460679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.460628 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:56.682973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.682890 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" Oct 30 07:16:56.683133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.682896 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84" event={"ID":"3d598ee3-6ee2-4a08-bd1a-8b9285d94078","Type":"ContainerDied","Data":"b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3"} Oct 30 07:16:56.683133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.683032 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b92848e9d2b1755acd361bf4d7fdbfa64bc91df3e452d193bf280608c92115f3" Oct 30 07:16:56.685060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.685021 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" event={"ID":"6b2f4984-85ca-4da3-9d66-9de09c972a8c","Type":"ContainerStarted","Data":"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce"} Oct 30 07:16:56.685218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.685068 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" event={"ID":"6b2f4984-85ca-4da3-9d66-9de09c972a8c","Type":"ContainerStarted","Data":"ac9caa636378dceda6c76a00d64abf29a48eff4b9ea54b5c0609ed2023dfeef0"} Oct 30 07:16:56.735381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.735259 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podStartSLOduration=1.7352397819999998 podStartE2EDuration="1.735239782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:56.733740631 +0000 UTC m=+1180.424658069" watchObservedRunningTime="2026-10-30 07:16:56.735239782 +0000 UTC m=+1180.426157222" Oct 30 07:16:56.773983 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:16:56.773952 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb14ff36_0d1b_4498_a8ae_3954d10cfdc7.slice/crio-e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534 WatchSource:0}: Error finding container e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534: Status 404 returned error can't find the container with id e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534 Oct 30 07:16:56.775619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.775588 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm"] Oct 30 07:16:56.920288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:56.920254 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:16:57.705446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.705412 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" event={"ID":"db14ff36-0d1b-4498-a8ae-3954d10cfdc7","Type":"ContainerStarted","Data":"01bf48a1142a617c18f5fe31c2454f0918ca7c8d47d15fd3196e4017a90539f1"} Oct 30 07:16:57.705878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.705468 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" event={"ID":"db14ff36-0d1b-4498-a8ae-3954d10cfdc7","Type":"ContainerStarted","Data":"e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534"} Oct 30 07:16:57.752151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.752091 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" podStartSLOduration=1.7520758829999998 podStartE2EDuration="1.752075883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:16:57.749700965 +0000 UTC m=+1181.440618401" watchObservedRunningTime="2026-10-30 07:16:57.752075883 +0000 UTC m=+1181.442993319" Oct 30 07:16:57.899823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.899772 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:57.926524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.926391 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 07:16:57.926789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:57.926587 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 07:16:58.107925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:58.107887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:58.714851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:58.714818 5894 generic.go:356] "Generic (PLEG): container finished" podID="db14ff36-0d1b-4498-a8ae-3954d10cfdc7" containerID="01bf48a1142a617c18f5fe31c2454f0918ca7c8d47d15fd3196e4017a90539f1" exitCode=0 Oct 30 07:16:58.715335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:58.714913 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" event={"ID":"db14ff36-0d1b-4498-a8ae-3954d10cfdc7","Type":"ContainerDied","Data":"01bf48a1142a617c18f5fe31c2454f0918ca7c8d47d15fd3196e4017a90539f1"} Oct 30 07:16:58.920335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:58.920294 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 07:16:58.920512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:58.920352 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 07:16:59.437292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.437262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:16:59.911110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.911085 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:16:59.919950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.919922 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 07:16:59.920096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.919992 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 07:16:59.939794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.939762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\") pod \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " Oct 30 07:16:59.939966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.939809 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\" (UniqueName: \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\") pod \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " Oct 30 07:16:59.939966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.939866 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\") pod \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\" (UID: \"db14ff36-0d1b-4498-a8ae-3954d10cfdc7\") " Oct 30 07:16:59.940193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.940166 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca" pod "db14ff36-0d1b-4498-a8ae-3954d10cfdc7" (UID: "db14ff36-0d1b-4498-a8ae-3954d10cfdc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:16:59.940516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.940498 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:16:59.941965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.941941 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx" pod "db14ff36-0d1b-4498-a8ae-3954d10cfdc7" (UID: "db14ff36-0d1b-4498-a8ae-3954d10cfdc7"). InnerVolumeSpecName "kube-api-access-k26jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:16:59.942060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:16:59.942036 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca" pod "db14ff36-0d1b-4498-a8ae-3954d10cfdc7" (UID: "db14ff36-0d1b-4498-a8ae-3954d10cfdc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:00.041998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.041963 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k26jx\" (UniqueName: \"kubernetes.io/projected/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-kube-api-access-k26jx\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:00.041998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.041994 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db14ff36-0d1b-4498-a8ae-3954d10cfdc7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:00.732543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.732461 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" Oct 30 07:17:00.732543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.732496 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm" event={"ID":"db14ff36-0d1b-4498-a8ae-3954d10cfdc7","Type":"ContainerDied","Data":"e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534"} Oct 30 07:17:00.732543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.732544 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e980db3eff36efb528ce4f1fba9e005ae9410562ba9fa51049a89f6233927534" Oct 30 07:17:00.919858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.919819 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 07:17:00.920335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:00.919882 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 07:17:01.097042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:01.097003 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:01.920023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:01.919988 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 07:17:01.920447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:01.920043 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 07:17:02.697652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:02.697612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:02.925117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:02.925084 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:17:02.925513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:02.925460 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:17:02.926108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:02.926088 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:17:03.542902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:03.542865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:03.713318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:03.713276 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:03.995712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:03.995597 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.144059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:04.144023 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.307804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:04.307759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.441617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:04.441585 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.606477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:04.606390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.792701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:04.792667 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:04.834586 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:17:04.834549 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:17:06.013544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:06.013496 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:06.782319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:06.782282 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:07.017184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:07.017151 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:07.500208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:07.500171 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:17:07.714792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:07.714760 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:08.699651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:08.699606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:08.829544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:08.829505 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:09.608504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:09.608472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:10.402258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.402220 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:10.518198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.518161 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:10.588091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.588047 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:17:10.588788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.588768 5894 cpu_manager.go:414] "containerMap: removing container" podUID="db14ff36-0d1b-4498-a8ae-3954d10cfdc7" containerName="revoke-keys" Oct 30 07:17:10.588788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.588788 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db14ff36-0d1b-4498-a8ae-3954d10cfdc7" containerName="revoke-keys" Oct 30 07:17:10.588955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.588885 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="db14ff36-0d1b-4498-a8ae-3954d10cfdc7" containerName="revoke-keys" Oct 30 07:17:10.592900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.592881 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.612503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.612471 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:17:10.619200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.619172 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6cc5f2-dockercfg-dstwb\"" Oct 30 07:17:10.620066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.620036 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6cc5f2\"" Oct 30 07:17:10.661587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.659985 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.662412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.662379 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8j6b\" (UniqueName: \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.672071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.672014 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:17:10.677902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.677873 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.759739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.759688 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:17:10.765369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.765333 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.766545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.766521 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvgn\" (UniqueName: \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.766899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.766880 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.767093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.766449 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.767377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.767354 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8j6b\" (UniqueName: \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.832247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.832214 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8j6b\" (UniqueName: \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\") pod \"payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:10.871647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.870927 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvgn\" (UniqueName: \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.871647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.871084 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.873504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.872480 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.899872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.899836 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvgn\" (UniqueName: \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\") pod \"payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:10.908917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:10.908847 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:11.065077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.065035 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:11.223551 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:11.223509 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6ed6e2_5ed7_4758_a337_0e2c6883859a.slice/crio-bf1722ef8cd32d63267d56ef296dd7edde53d29455341fc3aaf06d45d6f90435 WatchSource:0}: Error finding container bf1722ef8cd32d63267d56ef296dd7edde53d29455341fc3aaf06d45d6f90435: Status 404 returned error can't find the container with id bf1722ef8cd32d63267d56ef296dd7edde53d29455341fc3aaf06d45d6f90435 Oct 30 07:17:11.225172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.225147 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:17:11.379394 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:11.379363 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5c646d_b7eb_4eec_a8e7_54e723b42a4a.slice/crio-59287821fb8ce0696d4ad489aba0a89837eb38db331d7d984dda3887e9e7a894 WatchSource:0}: Error finding container 59287821fb8ce0696d4ad489aba0a89837eb38db331d7d984dda3887e9e7a894: Status 404 returned error can't find the container with id 59287821fb8ce0696d4ad489aba0a89837eb38db331d7d984dda3887e9e7a894 Oct 30 07:17:11.379979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.379953 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:17:11.417379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.417345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:11.847475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.847424 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" event={"ID":"ce6ed6e2-5ed7-4758-a337-0e2c6883859a","Type":"ContainerStarted","Data":"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b"} Oct 30 07:17:11.847734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.847509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" event={"ID":"ce6ed6e2-5ed7-4758-a337-0e2c6883859a","Type":"ContainerStarted","Data":"bf1722ef8cd32d63267d56ef296dd7edde53d29455341fc3aaf06d45d6f90435"} Oct 30 07:17:11.847734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.847539 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:11.849410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.849378 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" event={"ID":"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a","Type":"ContainerStarted","Data":"7a25c2a0f72c8db4f8cfc33d9313d82e8452e504c147d271b980e2f670fb2f2a"} Oct 30 07:17:11.849553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.849431 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:11.849553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.849448 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" event={"ID":"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a","Type":"ContainerStarted","Data":"59287821fb8ce0696d4ad489aba0a89837eb38db331d7d984dda3887e9e7a894"} Oct 30 07:17:11.894449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.894174 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" podStartSLOduration=1.894156222 podStartE2EDuration="1.894156222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:11.888084342 +0000 UTC m=+1195.579001780" watchObservedRunningTime="2026-10-30 07:17:11.894156222 +0000 UTC m=+1195.585073661" Oct 30 07:17:11.931303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:11.931251 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" podStartSLOduration=1.931237443 podStartE2EDuration="1.931237443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:11.928407072 +0000 UTC m=+1195.619324508" watchObservedRunningTime="2026-10-30 07:17:11.931237443 +0000 UTC m=+1195.622154879" Oct 30 07:17:13.037656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:13.035549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:14.853049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:14.853013 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:16.029596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.029563 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:16.238066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.238017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:16.713762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.713726 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:16.799935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.799905 5894 scope.go:122] "RemoveContainer" containerID="8dac2972b9d3383a273dc5ec135cf5cfcadb2f823cfda97b092ac1da5913e619" Oct 30 07:17:16.810644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.810614 5894 scope.go:122] "RemoveContainer" containerID="d47bcd0ffb603fe7bee5aad7979c6d1c223669e3fd74e3c5cbab1084486a3ed9" Oct 30 07:17:16.823042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.823013 5894 scope.go:122] "RemoveContainer" containerID="6f4d1bae568feec5770a2adc8f1a108e99c4b332cea3d1e78f18ff2936e68044" Oct 30 07:17:16.833208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.833183 5894 scope.go:122] "RemoveContainer" containerID="c2d3e2f91148883d66d19dcc3861597d74c856e82a62f8410886dc06e485eecf" Oct 30 07:17:16.843413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.843382 5894 scope.go:122] "RemoveContainer" containerID="ed25c96c1faf6fb02a146bf2e277811c1efcb448ee225149fbce342c7f51d5a3" Oct 30 07:17:16.853856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.853829 5894 scope.go:122] "RemoveContainer" containerID="0078a117151a0b3582dcf42270b47fe6332c5413d5fc370909240e6026b9ee6c" Oct 30 07:17:16.863895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:16.863867 5894 scope.go:122] "RemoveContainer" containerID="f3776ba761b79397f4aabb9cde3503c7f8498645ee7de2df1b84213f1f777c2d" Oct 30 07:17:17.182248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:17.182200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:17.387087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:17.387055 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:17.888959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:17.888921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:19.670147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:19.670105 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:21.032928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:21.032894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:21.167340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:21.167298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:21.446231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:21.446144 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:22.733442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.733407 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:22.855498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.855457 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:22.878335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.878307 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:17:22.878651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.878613 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:17:22.916452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.916419 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:17:22.939083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.939037 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:17:22.939272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.939173 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:22.958063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.958037 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-466248\"" Oct 30 07:17:22.958278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:22.958037 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-466248-dockercfg-xbd4n\"" Oct 30 07:17:23.009452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.009368 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:17:23.014118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.014083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:23.015734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.015625 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.038222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.038188 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:17:23.062206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.062170 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.062429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.062403 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv5dn\" (UniqueName: \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.176188 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.176326 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv5dn\" (UniqueName: \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.176477 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c29xc\" (UniqueName: \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.176548 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.177707 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.212508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.212443 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:23.232421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.232382 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv5dn\" (UniqueName: \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\") pod \"payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.263614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.260323 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.280673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.279263 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c29xc\" (UniqueName: \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.280673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.279464 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.280923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.280816 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.312684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.310670 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c29xc\" (UniqueName: \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\") pod \"payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.335770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.335735 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:23.600304 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:23.600276 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6201fa9_d8c5_4eb8_a037_3d34c8391c7f.slice/crio-4c730fef951316255036ad46dcfec7d6e6e7288281c63188e8937332afd04420 WatchSource:0}: Error finding container 4c730fef951316255036ad46dcfec7d6e6e7288281c63188e8937332afd04420: Status 404 returned error can't find the container with id 4c730fef951316255036ad46dcfec7d6e6e7288281c63188e8937332afd04420 Oct 30 07:17:23.601964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.601940 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:17:23.673773 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:23.673742 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9eeaae2_60d5_47b8_811c_2bb6814035f3.slice/crio-7632e08712fa4eaf74ac0db1232e234fcfa7b5353b9f702841bd81165d2aceb0 WatchSource:0}: Error finding container 7632e08712fa4eaf74ac0db1232e234fcfa7b5353b9f702841bd81165d2aceb0: Status 404 returned error can't find the container with id 7632e08712fa4eaf74ac0db1232e234fcfa7b5353b9f702841bd81165d2aceb0 Oct 30 07:17:23.674113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.674090 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:17:23.973284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.973087 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" event={"ID":"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f","Type":"ContainerStarted","Data":"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168"} Oct 30 07:17:23.973284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.973184 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" event={"ID":"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f","Type":"ContainerStarted","Data":"4c730fef951316255036ad46dcfec7d6e6e7288281c63188e8937332afd04420"} Oct 30 07:17:23.973845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.973347 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:23.975624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.975597 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" event={"ID":"c9eeaae2-60d5-47b8-811c-2bb6814035f3","Type":"ContainerStarted","Data":"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4"} Oct 30 07:17:23.975760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.975675 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" event={"ID":"c9eeaae2-60d5-47b8-811c-2bb6814035f3","Type":"ContainerStarted","Data":"7632e08712fa4eaf74ac0db1232e234fcfa7b5353b9f702841bd81165d2aceb0"} Oct 30 07:17:23.976292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:23.976275 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:24.022474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:24.022369 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" podStartSLOduration=2.022348745 podStartE2EDuration="2.022348745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:24.018107036 +0000 UTC m=+1207.709024473" watchObservedRunningTime="2026-10-30 07:17:24.022348745 +0000 UTC m=+1207.713266183" Oct 30 07:17:24.091384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:24.091330 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" podStartSLOduration=2.091312323 podStartE2EDuration="2.091312323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:24.086578823 +0000 UTC m=+1207.777496265" watchObservedRunningTime="2026-10-30 07:17:24.091312323 +0000 UTC m=+1207.782229759" Oct 30 07:17:24.833876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:24.833838 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:25.033286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:25.033253 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z"] Oct 30 07:17:25.047897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:25.047866 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbf139c6-lph5z"] Oct 30 07:17:26.433114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:26.433083 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2434627b-8a78-4863-a218-9c4f6363fc78" path="/var/lib/kubelet/pods/2434627b-8a78-4863-a218-9c4f6363fc78/volumes" Oct 30 07:17:26.553164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:26.553127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:26.670709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:26.670674 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:28.198528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:28.198488 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:29.879155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:29.879117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:31.506990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:31.506956 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:33.249747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:33.249708 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:34.033252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.033216 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw"] Oct 30 07:17:34.047467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.047432 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8efd01-dxffw"] Oct 30 07:17:34.393023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.392929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:34.433342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.433303 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11947bc-970f-46d5-b37b-b8b10746dac2" path="/var/lib/kubelet/pods/a11947bc-970f-46d5-b37b-b8b10746dac2/volumes" Oct 30 07:17:34.706124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.706035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:34.831180 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:17:34.831140 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:17:34.989317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:34.989241 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:17:35.269712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.261823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:35.389159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.389126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:35.416284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.416237 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm"] Oct 30 07:17:35.424527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.424493 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7c51aa-pz6fm"] Oct 30 07:17:35.521749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.520965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:35.946105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:35.946069 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:36.002259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:36.002229 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:17:36.435883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:36.435851 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db14ff36-0d1b-4498-a8ae-3954d10cfdc7" path="/var/lib/kubelet/pods/db14ff36-0d1b-4498-a8ae-3954d10cfdc7/volumes" Oct 30 07:17:36.972582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:36.972548 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:37.344262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:37.344224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:38.008832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.008796 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:38.132912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.132877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:38.144526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.144427 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84"] Oct 30 07:17:38.160193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.160154 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-77ef52-s4s84"] Oct 30 07:17:38.243770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.243710 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:38.432811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.432735 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d598ee3-6ee2-4a08-bd1a-8b9285d94078" path="/var/lib/kubelet/pods/3d598ee3-6ee2-4a08-bd1a-8b9285d94078/volumes" Oct 30 07:17:38.658188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:38.658148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:39.956850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:39.956817 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:40.036307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.036271 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn"] Oct 30 07:17:40.049079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.049038 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-060d7f6e-tgzwn"] Oct 30 07:17:40.081037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.081005 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:40.193579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.193544 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:40.434430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.434399 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c652383-ffbd-404b-8887-4a4eac003599" path="/var/lib/kubelet/pods/2c652383-ffbd-404b-8887-4a4eac003599/volumes" Oct 30 07:17:40.434761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.434745 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:17:40.441050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.441023 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.478749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.478716 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:17:40.597137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597095 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597149 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597174 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597355 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597550 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597684 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597748 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597812 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nt8p\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.597911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.597855 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.699696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.699602 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.699696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.699674 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.699696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.699696 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.699986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.699719 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nt8p\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.699986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.699747 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700102 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700137 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700164 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700329 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700433 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700700 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700115 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.700982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.700821 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.703435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.702476 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.703435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.702498 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.736589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.736518 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nt8p\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.737370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.737344 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\") pod \"e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:40.760023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:40.759734 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:41.124120 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:41.124086 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc720d10e_e4b8_4ab3_9c6a_eec069012b74.slice/crio-277f85d23958bc3b9922e779d726a3494cf1bc8e73bdc95560ac3b4d730bc26b WatchSource:0}: Error finding container 277f85d23958bc3b9922e779d726a3494cf1bc8e73bdc95560ac3b4d730bc26b: Status 404 returned error can't find the container with id 277f85d23958bc3b9922e779d726a3494cf1bc8e73bdc95560ac3b4d730bc26b Oct 30 07:17:41.125948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.125916 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:17:41.127774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.127742 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.127857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.127809 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.127857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.127838 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:17:41.158154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.158117 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" event={"ID":"c720d10e-e4b8-4ab3-9c6a-eec069012b74","Type":"ContainerStarted","Data":"277f85d23958bc3b9922e779d726a3494cf1bc8e73bdc95560ac3b4d730bc26b"} Oct 30 07:17:41.717294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:41.717264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:42.167698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.167610 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" event={"ID":"c720d10e-e4b8-4ab3-9c6a-eec069012b74","Type":"ContainerStarted","Data":"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d"} Oct 30 07:17:42.299498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.299428 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podStartSLOduration=2.299411051 podStartE2EDuration="2.299411051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:42.294878156 +0000 UTC m=+1225.985795594" watchObservedRunningTime="2026-10-30 07:17:42.299411051 +0000 UTC m=+1225.990328487" Oct 30 07:17:42.593303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.593268 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:42.713408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.713377 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:42.760008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.759971 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:42.761550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.761522 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 07:17:42.761723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.761583 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 07:17:42.952476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:42.952381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:43.646472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.646435 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d"] Oct 30 07:17:43.651229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.651209 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.670261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.670220 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d"] Oct 30 07:17:43.686664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.676606 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:17:43.686664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.676707 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:17:43.707806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.704725 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:43.709281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.709250 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:17:43.709584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.709549 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:17:43.760502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.760468 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 07:17:43.760946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.760915 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 07:17:43.783718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.778943 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.783718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.779269 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5gsq\" (UniqueName: \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.783718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.779361 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.832233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.832190 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:43.897773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.897578 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.897773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.897687 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.898004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.897951 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5gsq\" (UniqueName: \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.899395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.899366 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.901904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.901882 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.924096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.924069 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5gsq\" (UniqueName: \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\") pod \"maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:43.966472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:43.966438 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:44.009659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.008785 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:44.019540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.016487 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:17:44.019540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.016745 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" containerID="cri-o://7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662" gracePeriod=30 Oct 30 07:17:44.141660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.141208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:44.618143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.617918 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d"] Oct 30 07:17:44.760095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.760064 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 07:17:44.760695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.760120 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 07:17:44.953438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:44.953333 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:45.033558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.033525 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p"] Oct 30 07:17:45.038351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.038325 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f58211-r8d6p"] Oct 30 07:17:45.331181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.330231 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" event={"ID":"51172e93-621d-47a9-88df-23da73100a44","Type":"ContainerStarted","Data":"60f367a3ce6911c49533a837aff494a48889d8d3684e80d35ca61393a37755c4"} Oct 30 07:17:45.331181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.330281 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" event={"ID":"51172e93-621d-47a9-88df-23da73100a44","Type":"ContainerStarted","Data":"e2c0e1bf67d29ab76a59bcac1493eda88ea4ede667760132ae4edb675760ca7d"} Oct 30 07:17:45.341624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.341593 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:45.442665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.440442 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" podStartSLOduration=2.440422496 podStartE2EDuration="2.440422496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:45.420976 +0000 UTC m=+1229.111893439" watchObservedRunningTime="2026-10-30 07:17:45.440422496 +0000 UTC m=+1229.131339933" Oct 30 07:17:45.761339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.760833 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 07:17:45.761339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:45.760890 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 07:17:46.031660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.030320 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:46.169909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.169872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:46.291405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.291222 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:46.371050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.363199 5894 generic.go:356] "Generic (PLEG): container finished" podID="51172e93-621d-47a9-88df-23da73100a44" containerID="60f367a3ce6911c49533a837aff494a48889d8d3684e80d35ca61393a37755c4" exitCode=0 Oct 30 07:17:46.371050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.363339 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" event={"ID":"51172e93-621d-47a9-88df-23da73100a44","Type":"ContainerDied","Data":"60f367a3ce6911c49533a837aff494a48889d8d3684e80d35ca61393a37755c4"} Oct 30 07:17:46.431414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.431230 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:46.449207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.442616 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c64eea-e874-4ae2-a6db-61f394284ee3" path="/var/lib/kubelet/pods/b0c64eea-e874-4ae2-a6db-61f394284ee3/volumes" Oct 30 07:17:46.760465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.760363 5894 patch_prober.go:29] interesting pod/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 07:17:46.760465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.760417 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 07:17:46.926382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.926347 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564"] Oct 30 07:17:46.931020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.931002 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:46.947694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:46.947663 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564"] Oct 30 07:17:47.105659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.104982 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b896\" (UniqueName: \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.105659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.105319 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.105659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.105372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.206393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.206351 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b896\" (UniqueName: \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.206607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.206460 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.206607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.206508 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.207128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.207100 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.208946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.208925 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.233317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.233277 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b896\" (UniqueName: \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\") pod \"maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.248061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.248025 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:47.711608 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:47.711565 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6001970_d781_411b_950b_f4f391e7b93c.slice/crio-a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183 WatchSource:0}: Error finding container a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183: Status 404 returned error can't find the container with id a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183 Oct 30 07:17:47.713587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.713556 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564"] Oct 30 07:17:47.725015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.724975 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:47.822992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.822952 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\") pod \"51172e93-621d-47a9-88df-23da73100a44\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " Oct 30 07:17:47.823182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.823081 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\") pod \"51172e93-621d-47a9-88df-23da73100a44\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " Oct 30 07:17:47.823182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.823166 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\" (UniqueName: \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\") pod \"51172e93-621d-47a9-88df-23da73100a44\" (UID: \"51172e93-621d-47a9-88df-23da73100a44\") " Oct 30 07:17:47.824078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.824042 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca" pod "51172e93-621d-47a9-88df-23da73100a44" (UID: "51172e93-621d-47a9-88df-23da73100a44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:47.826222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.826177 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca" pod "51172e93-621d-47a9-88df-23da73100a44" (UID: "51172e93-621d-47a9-88df-23da73100a44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:47.826407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.826385 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq" pod "51172e93-621d-47a9-88df-23da73100a44" (UID: "51172e93-621d-47a9-88df-23da73100a44"). InnerVolumeSpecName "kube-api-access-t5gsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:47.925339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.925295 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5gsq\" (UniqueName: \"kubernetes.io/projected/51172e93-621d-47a9-88df-23da73100a44-kube-api-access-t5gsq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:47.925339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.925339 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51172e93-621d-47a9-88df-23da73100a44-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:47.925649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.925357 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51172e93-621d-47a9-88df-23da73100a44-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:47.945936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.945862 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:47.946408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.946357 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:47.947226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:47.947208 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:17:48.409166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.409136 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" Oct 30 07:17:48.409370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.409181 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d" event={"ID":"51172e93-621d-47a9-88df-23da73100a44","Type":"ContainerDied","Data":"e2c0e1bf67d29ab76a59bcac1493eda88ea4ede667760132ae4edb675760ca7d"} Oct 30 07:17:48.409370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.409238 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2c0e1bf67d29ab76a59bcac1493eda88ea4ede667760132ae4edb675760ca7d" Oct 30 07:17:48.411230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.411199 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" event={"ID":"f6001970-d781-411b-950b-f4f391e7b93c","Type":"ContainerStarted","Data":"60b93ee319fb3db348620489238f835b4967d987698d29e5c27d9c5d819cc1d0"} Oct 30 07:17:48.411369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.411250 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" event={"ID":"f6001970-d781-411b-950b-f4f391e7b93c","Type":"ContainerStarted","Data":"a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183"} Oct 30 07:17:48.467279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.467229 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" podStartSLOduration=2.467214405 podStartE2EDuration="2.467214405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:48.46662871 +0000 UTC m=+1232.157546149" watchObservedRunningTime="2026-10-30 07:17:48.467214405 +0000 UTC m=+1232.158131842" Oct 30 07:17:48.467561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:48.467538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:49.242456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.242265 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:49.325232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.325206 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:17:49.425582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.425544 5894 generic.go:356] "Generic (PLEG): container finished" podID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerID="7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662" exitCode=0 Oct 30 07:17:49.425828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.425604 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" Oct 30 07:17:49.425828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.425628 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" event={"ID":"75992f34-ee68-4ed4-bcf3-9d5209a52097","Type":"ContainerDied","Data":"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662"} Oct 30 07:17:49.425828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.425732 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn" event={"ID":"75992f34-ee68-4ed4-bcf3-9d5209a52097","Type":"ContainerDied","Data":"557da1e6d5814c4915759132c87de8405dd662fcfd6a201e8b740bd0335740e7"} Oct 30 07:17:49.425828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.425766 5894 scope.go:122] "RemoveContainer" containerID="7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662" Oct 30 07:17:49.428167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.428137 5894 generic.go:356] "Generic (PLEG): container finished" podID="f6001970-d781-411b-950b-f4f391e7b93c" containerID="60b93ee319fb3db348620489238f835b4967d987698d29e5c27d9c5d819cc1d0" exitCode=0 Oct 30 07:17:49.428284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.428233 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" event={"ID":"f6001970-d781-411b-950b-f4f391e7b93c","Type":"ContainerDied","Data":"60b93ee319fb3db348620489238f835b4967d987698d29e5c27d9c5d819cc1d0"} Oct 30 07:17:49.438084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.438063 5894 scope.go:122] "RemoveContainer" containerID="7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662" Oct 30 07:17:49.438438 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:17:49.438418 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662\": container with ID starting with 7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662 not found: ID does not exist" containerID="7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662" Oct 30 07:17:49.438541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.438448 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662"} err="failed to get container status \"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662\": rpc error: code = NotFound desc = could not find container \"7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662\": container with ID starting with 7f4b2063e0f4d06f235a9423f29c2028853df24450c5e1bdf248f889f2b27662 not found: ID does not exist" Oct 30 07:17:49.447049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447022 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447062 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447211 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447243 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447320 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.447404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447388 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447439 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447500 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.447599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447586 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447620 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\") pod \"75992f34-ee68-4ed4-bcf3-9d5209a52097\" (UID: \"75992f34-ee68-4ed4-bcf3-9d5209a52097\") " Oct 30 07:17:49.447897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447860 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.447897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447882 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:49.448031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.447985 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.448264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.448246 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75992f34-ee68-4ed4-bcf3-9d5209a52097-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.448264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.448264 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.448397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.448278 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.448397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.448290 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.448397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.448301 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.449402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.449379 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:17:49.449900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.449875 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:49.450009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.449913 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "kube-api-access-jm8xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:49.450009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.449986 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo" pod "75992f34-ee68-4ed4-bcf3-9d5209a52097" (UID: "75992f34-ee68-4ed4-bcf3-9d5209a52097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:17:49.550595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.550507 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.550595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.550543 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.550595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.550561 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.550595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.550578 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm8xt\" (UniqueName: \"kubernetes.io/projected/75992f34-ee68-4ed4-bcf3-9d5209a52097-kube-api-access-jm8xt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:49.593793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.593735 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:49.729363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.729328 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:49.781684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.781646 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:17:49.802396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.802323 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d0dace8c-openshift-default-85c679b469-ml9dn"] Oct 30 07:17:49.917364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:49.917322 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:50.044301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.044244 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx"] Oct 30 07:17:50.048990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.048948 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86cd2e7b-8m4bx"] Oct 30 07:17:50.440963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.440921 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" path="/var/lib/kubelet/pods/75992f34-ee68-4ed4-bcf3-9d5209a52097/volumes" Oct 30 07:17:50.441689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.441627 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7" path="/var/lib/kubelet/pods/a85f1f9d-cc0e-4639-9cd8-3d1f63355cd7/volumes" Oct 30 07:17:50.443323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.443286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:50.571531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.571498 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:50.763729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.759567 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.779794 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\" (UniqueName: \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\") pod \"f6001970-d781-411b-950b-f4f391e7b93c\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.779868 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\") pod \"f6001970-d781-411b-950b-f4f391e7b93c\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.779996 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\") pod \"f6001970-d781-411b-950b-f4f391e7b93c\" (UID: \"f6001970-d781-411b-950b-f4f391e7b93c\") " Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.780289 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca" pod "f6001970-d781-411b-950b-f4f391e7b93c" (UID: "f6001970-d781-411b-950b-f4f391e7b93c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.780713 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.782482 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca" pod "f6001970-d781-411b-950b-f4f391e7b93c" (UID: "f6001970-d781-411b-950b-f4f391e7b93c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:17:50.787977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.783436 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896" pod "f6001970-d781-411b-950b-f4f391e7b93c" (UID: "f6001970-d781-411b-950b-f4f391e7b93c"). InnerVolumeSpecName "kube-api-access-7b896". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:17:50.882256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.882218 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6001970-d781-411b-950b-f4f391e7b93c-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:50.882256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:50.882258 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b896\" (UniqueName: \"kubernetes.io/projected/f6001970-d781-411b-950b-f4f391e7b93c-kube-api-access-7b896\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:17:51.439724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.439672 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:51.459588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.459548 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" event={"ID":"f6001970-d781-411b-950b-f4f391e7b93c","Type":"ContainerDied","Data":"a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183"} Oct 30 07:17:51.460200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.460168 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83e124f362c7992ab65fe8c7ac0d61c462e437de14df515b0cb95715b1e6183" Oct 30 07:17:51.460330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.460260 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564" Oct 30 07:17:51.562416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.562383 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:17:51.563179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563153 5894 cpu_manager.go:414] "containerMap: removing container" podUID="51172e93-621d-47a9-88df-23da73100a44" containerName="revoke-keys" Oct 30 07:17:51.563179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563178 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51172e93-621d-47a9-88df-23da73100a44" containerName="revoke-keys" Oct 30 07:17:51.563364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563206 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f6001970-d781-411b-950b-f4f391e7b93c" containerName="revoke-keys" Oct 30 07:17:51.563364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563216 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6001970-d781-411b-950b-f4f391e7b93c" containerName="revoke-keys" Oct 30 07:17:51.563364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563234 5894 cpu_manager.go:414] "containerMap: removing container" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" Oct 30 07:17:51.563364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563241 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" Oct 30 07:17:51.563508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563393 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="51172e93-621d-47a9-88df-23da73100a44" containerName="revoke-keys" Oct 30 07:17:51.563508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563404 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6001970-d781-411b-950b-f4f391e7b93c" containerName="revoke-keys" Oct 30 07:17:51.563508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.563412 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="75992f34-ee68-4ed4-bcf3-9d5209a52097" containerName="istio-proxy" Oct 30 07:17:51.566805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.566771 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:51.567564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.567547 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.587931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.587900 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:17:51.591040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.590151 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4c43f7\"" Oct 30 07:17:51.593141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.593108 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dlft\" (UniqueName: \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.593282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.593262 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.606191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.605485 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:17:51.610793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.610767 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.629183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.628744 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4c43f7-dockercfg-8j7dv\"" Oct 30 07:17:51.639696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.639591 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:17:51.706715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.703489 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.706715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.703580 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nsqx\" (UniqueName: \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.706715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.703757 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dlft\" (UniqueName: \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.706715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.703926 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.706715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.705446 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.769662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.761697 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dlft\" (UniqueName: \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\") pod \"payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.815690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.812889 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.815690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.812976 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nsqx\" (UniqueName: \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.815690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.814470 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.845441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.844375 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nsqx\" (UniqueName: \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\") pod \"payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:51.885757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.885538 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:51.941989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:51.941959 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:52.251570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.251535 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:17:52.261186 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:52.261154 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222e7790_9c7a_4f88_a77d_96ce89ccd617.slice/crio-d93c9e92bd33b8f11e4accd21bcf1ebac8a20156e51b9e34417d779754a22790 WatchSource:0}: Error finding container d93c9e92bd33b8f11e4accd21bcf1ebac8a20156e51b9e34417d779754a22790: Status 404 returned error can't find the container with id d93c9e92bd33b8f11e4accd21bcf1ebac8a20156e51b9e34417d779754a22790 Oct 30 07:17:52.309607 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:17:52.309575 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2785b3_cc84_4e5a_92d3_115968622fb9.slice/crio-c99dc79df3f4fb994aaf90146cab53acf7893bd8d04c5cb17b543c863294383b WatchSource:0}: Error finding container c99dc79df3f4fb994aaf90146cab53acf7893bd8d04c5cb17b543c863294383b: Status 404 returned error can't find the container with id c99dc79df3f4fb994aaf90146cab53acf7893bd8d04c5cb17b543c863294383b Oct 30 07:17:52.310920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.310892 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:17:52.504775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.504679 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" event={"ID":"5c2785b3-cc84-4e5a-92d3-115968622fb9","Type":"ContainerStarted","Data":"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373"} Oct 30 07:17:52.504775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.504753 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" event={"ID":"5c2785b3-cc84-4e5a-92d3-115968622fb9","Type":"ContainerStarted","Data":"c99dc79df3f4fb994aaf90146cab53acf7893bd8d04c5cb17b543c863294383b"} Oct 30 07:17:52.505488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.504827 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:17:52.508167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.508136 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" event={"ID":"222e7790-9c7a-4f88-a77d-96ce89ccd617","Type":"ContainerStarted","Data":"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c"} Oct 30 07:17:52.508399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.508369 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" event={"ID":"222e7790-9c7a-4f88-a77d-96ce89ccd617","Type":"ContainerStarted","Data":"d93c9e92bd33b8f11e4accd21bcf1ebac8a20156e51b9e34417d779754a22790"} Oct 30 07:17:52.508529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.508421 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:17:52.513309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.513278 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:52.554987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.554896 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" podStartSLOduration=1.55487548 podStartE2EDuration="1.55487548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:52.548962815 +0000 UTC m=+1236.239880265" watchObservedRunningTime="2026-10-30 07:17:52.55487548 +0000 UTC m=+1236.245792919" Oct 30 07:17:52.603351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:52.603246 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" podStartSLOduration=1.603231317 podStartE2EDuration="1.603231317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:17:52.602987888 +0000 UTC m=+1236.293905336" watchObservedRunningTime="2026-10-30 07:17:52.603231317 +0000 UTC m=+1236.294148754" Oct 30 07:17:53.030878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:53.030841 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4"] Oct 30 07:17:53.034755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:53.034725 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c80d241-c68s4"] Oct 30 07:17:54.334856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:54.334040 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:54.433527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:54.433490 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984c2125-5e72-41b1-808f-bdd0ad4e6979" path="/var/lib/kubelet/pods/984c2125-5e72-41b1-808f-bdd0ad4e6979/volumes" Oct 30 07:17:54.451242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:54.451199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:54.695473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:54.695394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:54.904455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:54.904423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:55.051767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:55.051723 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:56.552266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:56.552230 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:56.669212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:56.669170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:57.034496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:57.034466 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb"] Oct 30 07:17:57.047033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:57.046999 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d8fddf-2vbtb"] Oct 30 07:17:57.302458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:57.302357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:57.415221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:57.415180 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:17:58.432713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:58.432675 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d7de361-83df-452e-8725-840cb63d2824" path="/var/lib/kubelet/pods/1d7de361-83df-452e-8725-840cb63d2824/volumes" Oct 30 07:17:58.552097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:17:58.552061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:00.439838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:00.439804 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:01.531735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:01.531701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:01.646453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:01.646418 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:02.232356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:02.232318 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:03.520373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:03.520344 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:18:03.520987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:03.520791 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:18:04.030717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:04.030677 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5"] Oct 30 07:18:04.034288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:04.034262 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c514af58-khbk5"] Oct 30 07:18:04.069100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:04.069069 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:04.432495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:04.432415 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d64b6301-19b8-4009-8854-77f1e00b7d2a" path="/var/lib/kubelet/pods/d64b6301-19b8-4009-8854-77f1e00b7d2a/volumes" Oct 30 07:18:04.828987 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:04.828952 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:18:05.234371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:05.234279 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:18:05.775936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:05.775879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:06.797387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:06.797337 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:07.038551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.038515 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn"] Oct 30 07:18:07.053046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.052948 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51c9f5aa-6gwdn"] Oct 30 07:18:07.581129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.581091 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:07.718295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.718262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:07.729323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.729223 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44"] Oct 30 07:18:07.752664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.743014 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.759937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.759892 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44"] Oct 30 07:18:07.768681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.762106 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:18:07.768681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.762226 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:18:07.796093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.796061 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:18:07.796477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.796458 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:18:07.807382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.805998 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.807382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.806172 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6ltb\" (UniqueName: \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.807382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.806262 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.907738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.907620 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.907939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.907774 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6ltb\" (UniqueName: \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.907939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.907862 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.908571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.908539 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.910195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.910169 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:07.932992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:07.932965 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6ltb\" (UniqueName: \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\") pod \"maas-api-revoke-keys-e2e-api-a-318524-wvq44\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:08.105829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:08.105798 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:08.436731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:08.436622 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5236f9-91d4-4017-9834-eeb242fa7799" path="/var/lib/kubelet/pods/ee5236f9-91d4-4017-9834-eeb242fa7799/volumes" Oct 30 07:18:08.470672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:08.469294 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44"] Oct 30 07:18:08.661129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:08.661092 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" event={"ID":"32f81a86-fc31-4954-b156-6e8d2bd5fa37","Type":"ContainerStarted","Data":"5ce76f5b569442f8180d52d6b5ffd446faac7d4eef802c3aee36fbb1a47eb395"} Oct 30 07:18:09.681248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:09.681160 5894 generic.go:356] "Generic (PLEG): container finished" podID="32f81a86-fc31-4954-b156-6e8d2bd5fa37" containerID="ddf9897fb244a56fc553c46731a4b0930f8454bfbf0b0301861eb97498d5ecbc" exitCode=0 Oct 30 07:18:09.681248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:09.681202 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" event={"ID":"32f81a86-fc31-4954-b156-6e8d2bd5fa37","Type":"ContainerDied","Data":"ddf9897fb244a56fc553c46731a4b0930f8454bfbf0b0301861eb97498d5ecbc"} Oct 30 07:18:09.770121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:09.770090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:10.857435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.857406 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:10.980401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.980307 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\") pod \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " Oct 30 07:18:10.980401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.980372 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\") pod \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " Oct 30 07:18:10.980609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.980468 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\" (UniqueName: \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\") pod \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\" (UID: \"32f81a86-fc31-4954-b156-6e8d2bd5fa37\") " Oct 30 07:18:10.980768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.980745 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca" pod "32f81a86-fc31-4954-b156-6e8d2bd5fa37" (UID: "32f81a86-fc31-4954-b156-6e8d2bd5fa37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:10.981259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.981230 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:10.982680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.982653 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca" pod "32f81a86-fc31-4954-b156-6e8d2bd5fa37" (UID: "32f81a86-fc31-4954-b156-6e8d2bd5fa37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:10.982829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:10.982748 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb" pod "32f81a86-fc31-4954-b156-6e8d2bd5fa37" (UID: "32f81a86-fc31-4954-b156-6e8d2bd5fa37"). InnerVolumeSpecName "kube-api-access-d6ltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:11.082545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.082513 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f81a86-fc31-4954-b156-6e8d2bd5fa37-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:11.082545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.082543 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6ltb\" (UniqueName: \"kubernetes.io/projected/32f81a86-fc31-4954-b156-6e8d2bd5fa37-kube-api-access-d6ltb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:11.684001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.683956 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:11.699135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.699107 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" Oct 30 07:18:11.699309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.699142 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44" event={"ID":"32f81a86-fc31-4954-b156-6e8d2bd5fa37","Type":"ContainerDied","Data":"5ce76f5b569442f8180d52d6b5ffd446faac7d4eef802c3aee36fbb1a47eb395"} Oct 30 07:18:11.699309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:11.699187 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce76f5b569442f8180d52d6b5ffd446faac7d4eef802c3aee36fbb1a47eb395" Oct 30 07:18:12.034030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:12.033999 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755"] Oct 30 07:18:12.037561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:12.037538 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8387e190-7p755"] Oct 30 07:18:12.433460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:12.433381 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f304254c-3904-4e9b-b52c-f49f1505c9f4" path="/var/lib/kubelet/pods/f304254c-3904-4e9b-b52c-f49f1505c9f4/volumes" Oct 30 07:18:13.493498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:13.493456 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:13.611746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:13.611705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:15.350806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:15.350768 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:16.030382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:16.030351 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr"] Oct 30 07:18:16.044567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:16.044531 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-add5574c-7stnr"] Oct 30 07:18:16.432434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:16.432345 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5930a586-bbc4-4e42-bbb9-606a4d276100" path="/var/lib/kubelet/pods/5930a586-bbc4-4e42-bbb9-606a4d276100/volumes" Oct 30 07:18:16.943442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:16.943411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:17.966788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:17.966753 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:18.086914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.086868 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:18.147567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.147534 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:18:18.182966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.182927 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:18:18.286207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.286167 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:18.396945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.396824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:18.750845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:18.750756 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:19.038739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:19.038705 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8"] Oct 30 07:18:19.042284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:19.042256 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-146b6361-c7vt8"] Oct 30 07:18:20.207924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.207877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:20.445852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.445816 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229d445f-11c6-4212-8e64-72b8da199703" path="/var/lib/kubelet/pods/229d445f-11c6-4212-8e64-72b8da199703/volumes" Oct 30 07:18:20.766987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.766950 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:20.886848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.886813 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:20.931736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.931704 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564"] Oct 30 07:18:20.945135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.945104 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-1e3dd1-ws564"] Oct 30 07:18:20.999752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:20.999716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:21.439151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:21.439110 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:22.032567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:22.032530 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl"] Oct 30 07:18:22.046003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:22.045973 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb75cf-c96cl"] Oct 30 07:18:22.433505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:22.433420 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0665c44-138b-419b-89ec-2906d62bf6ee" path="/var/lib/kubelet/pods/e0665c44-138b-419b-89ec-2906d62bf6ee/volumes" Oct 30 07:18:22.433938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:22.433921 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6001970-d781-411b-950b-f4f391e7b93c" path="/var/lib/kubelet/pods/f6001970-d781-411b-950b-f4f391e7b93c/volumes" Oct 30 07:18:22.736206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:22.736123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:23.148846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.148806 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" podUID="860e0667-394a-4cc8-93cc-f2694b5a178d" containerName="payload-processing" containerID="cri-o://a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03" gracePeriod=25 Oct 30 07:18:23.183444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.183398 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" podUID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" containerName="payload-pre-processing" containerID="cri-o://1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766" gracePeriod=25 Oct 30 07:18:23.472328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.471956 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:18:23.514894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.514861 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:18:23.599682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.599467 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\") pod \"860e0667-394a-4cc8-93cc-f2694b5a178d\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " Oct 30 07:18:23.599865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.599708 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\" (UniqueName: \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\") pod \"860e0667-394a-4cc8-93cc-f2694b5a178d\" (UID: \"860e0667-394a-4cc8-93cc-f2694b5a178d\") " Oct 30 07:18:23.599936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.599913 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume" pod "860e0667-394a-4cc8-93cc-f2694b5a178d" (UID: "860e0667-394a-4cc8-93cc-f2694b5a178d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:23.600780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.600582 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/860e0667-394a-4cc8-93cc-f2694b5a178d-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:23.601994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.601971 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz" pod "860e0667-394a-4cc8-93cc-f2694b5a178d" (UID: "860e0667-394a-4cc8-93cc-f2694b5a178d"). InnerVolumeSpecName "kube-api-access-tm7tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:23.701996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.701899 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\") pod \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " Oct 30 07:18:23.702173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.702103 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\" (UniqueName: \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\") pod \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\" (UID: \"abb4e048-b97d-4b33-b9ad-0f5c6d147b66\") " Oct 30 07:18:23.702275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.702250 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume" pod "abb4e048-b97d-4b33-b9ad-0f5c6d147b66" (UID: "abb4e048-b97d-4b33-b9ad-0f5c6d147b66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:23.702739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.702710 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:23.702739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.702738 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm7tz\" (UniqueName: \"kubernetes.io/projected/860e0667-394a-4cc8-93cc-f2694b5a178d-kube-api-access-tm7tz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:23.704255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.704235 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g" pod "abb4e048-b97d-4b33-b9ad-0f5c6d147b66" (UID: "abb4e048-b97d-4b33-b9ad-0f5c6d147b66"). InnerVolumeSpecName "kube-api-access-vqk2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:23.804190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.804143 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqk2g\" (UniqueName: \"kubernetes.io/projected/abb4e048-b97d-4b33-b9ad-0f5c6d147b66-kube-api-access-vqk2g\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:23.813446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.813416 5894 generic.go:356] "Generic (PLEG): container finished" podID="860e0667-394a-4cc8-93cc-f2694b5a178d" containerID="a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03" exitCode=0 Oct 30 07:18:23.813621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.813459 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" Oct 30 07:18:23.813621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.813503 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" event={"ID":"860e0667-394a-4cc8-93cc-f2694b5a178d","Type":"ContainerDied","Data":"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03"} Oct 30 07:18:23.813621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.813553 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz" event={"ID":"860e0667-394a-4cc8-93cc-f2694b5a178d","Type":"ContainerDied","Data":"56dc211f9282a361c95009bcfe6ba6c7e35fa415264e34c740da79aa1940b024"} Oct 30 07:18:23.813621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.813573 5894 scope.go:122] "RemoveContainer" containerID="a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03" Oct 30 07:18:23.815232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.815198 5894 generic.go:356] "Generic (PLEG): container finished" podID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" containerID="1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766" exitCode=0 Oct 30 07:18:23.815331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.815293 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" Oct 30 07:18:23.815331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.815317 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" event={"ID":"abb4e048-b97d-4b33-b9ad-0f5c6d147b66","Type":"ContainerDied","Data":"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766"} Oct 30 07:18:23.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.815356 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8" event={"ID":"abb4e048-b97d-4b33-b9ad-0f5c6d147b66","Type":"ContainerDied","Data":"fa2b3e16fa8685dc8a4cf16ee982ca70a6b4aedefc62f4cf0dcd97fe3d10a27b"} Oct 30 07:18:23.831536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.831518 5894 scope.go:122] "RemoveContainer" containerID="a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03" Oct 30 07:18:23.831914 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:23.831890 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03\": container with ID starting with a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03 not found: ID does not exist" containerID="a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03" Oct 30 07:18:23.831977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.831924 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03"} err="failed to get container status \"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03\": rpc error: code = NotFound desc = could not find container \"a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03\": container with ID starting with a7e10e5657c26993ce67361baabd7cf138f074cd3deea0747859c89fa3e4ba03 not found: ID does not exist" Oct 30 07:18:23.831977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.831942 5894 scope.go:122] "RemoveContainer" containerID="1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766" Oct 30 07:18:23.842184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.842163 5894 scope.go:122] "RemoveContainer" containerID="1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766" Oct 30 07:18:23.842508 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:23.842485 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766\": container with ID starting with 1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766 not found: ID does not exist" containerID="1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766" Oct 30 07:18:23.842570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.842513 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766"} err="failed to get container status \"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766\": rpc error: code = NotFound desc = could not find container \"1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766\": container with ID starting with 1db5763c33b271a7c4f1e8eda0ac3393017227ba010909ea9df3af12fef3f766 not found: ID does not exist" Oct 30 07:18:23.872808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.872769 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:18:23.886880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.886850 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-318524-66755578d-8zlrz"] Oct 30 07:18:23.917647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.917615 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:18:23.920909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:23.920886 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-318524-6f5678c75-bqfh8"] Oct 30 07:18:24.352294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:24.352262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:24.433986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:24.433889 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860e0667-394a-4cc8-93cc-f2694b5a178d" path="/var/lib/kubelet/pods/860e0667-394a-4cc8-93cc-f2694b5a178d/volumes" Oct 30 07:18:24.434764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:24.434729 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" path="/var/lib/kubelet/pods/abb4e048-b97d-4b33-b9ad-0f5c6d147b66/volumes" Oct 30 07:18:25.940120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:25.940086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:26.034690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:26.034623 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p"] Oct 30 07:18:26.047591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:26.047556 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c068d358-clk4p"] Oct 30 07:18:26.433186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:26.433155 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c271f8-333c-44d8-8633-259d55bb6347" path="/var/lib/kubelet/pods/e4c271f8-333c-44d8-8633-259d55bb6347/volumes" Oct 30 07:18:26.753583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:26.753466 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:27.613030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:27.612999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:28.209407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:28.209368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:28.343526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:28.343488 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:28.807043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:28.807011 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:29.320457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:29.320345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:30.065125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.065091 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:30.196871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.196838 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:30.220464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.220426 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d"] Oct 30 07:18:30.234933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.234899 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-56eb7f-hhj8d"] Oct 30 07:18:30.316119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.316027 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:30.433531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.433497 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51172e93-621d-47a9-88df-23da73100a44" path="/var/lib/kubelet/pods/51172e93-621d-47a9-88df-23da73100a44/volumes" Oct 30 07:18:30.747281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.747192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:30.868839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:30.868802 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:31.372194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:31.372049 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:31.590485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:31.590453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:32.242359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:32.242325 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:34.057597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:34.057556 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:34.829070 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:34.829031 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:18:35.346508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:35.346474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:35.802159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:35.802117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:35.948257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:35.948224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:36.138120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.138023 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:36.519074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519001 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z"] Oct 30 07:18:36.519681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519662 5894 cpu_manager.go:414] "containerMap: removing container" podUID="32f81a86-fc31-4954-b156-6e8d2bd5fa37" containerName="revoke-keys" Oct 30 07:18:36.519681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519679 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f81a86-fc31-4954-b156-6e8d2bd5fa37" containerName="revoke-keys" Oct 30 07:18:36.519790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519747 5894 cpu_manager.go:414] "containerMap: removing container" podUID="860e0667-394a-4cc8-93cc-f2694b5a178d" containerName="payload-processing" Oct 30 07:18:36.519790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519753 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860e0667-394a-4cc8-93cc-f2694b5a178d" containerName="payload-processing" Oct 30 07:18:36.519790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519787 5894 cpu_manager.go:414] "containerMap: removing container" podUID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" containerName="payload-pre-processing" Oct 30 07:18:36.519888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519809 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" containerName="payload-pre-processing" Oct 30 07:18:36.519921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519917 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="860e0667-394a-4cc8-93cc-f2694b5a178d" containerName="payload-processing" Oct 30 07:18:36.519951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519931 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="abb4e048-b97d-4b33-b9ad-0f5c6d147b66" containerName="payload-pre-processing" Oct 30 07:18:36.519951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.519938 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f81a86-fc31-4954-b156-6e8d2bd5fa37" containerName="revoke-keys" Oct 30 07:18:36.523372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.523353 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.543800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.543761 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z"] Oct 30 07:18:36.551106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.551076 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:18:36.551338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.551110 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:18:36.551338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.551164 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:18:36.551507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.551257 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:18:36.642730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.642690 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpf9r\" (UniqueName: \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.642984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.642964 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.643059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.643045 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.744503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.744457 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.744757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.744561 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.744836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.744780 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpf9r\" (UniqueName: \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.745274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.745252 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.747378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.747350 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.768861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.768828 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpf9r\" (UniqueName: \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\") pod \"maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:36.837863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:36.837828 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:37.149796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.149768 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z"] Oct 30 07:18:37.150026 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:18:37.149989 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1209cabb_e68a_4402_89a2_3a8cc5f84a38.slice/crio-8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e WatchSource:0}: Error finding container 8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e: Status 404 returned error can't find the container with id 8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e Oct 30 07:18:37.432986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.432908 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:37.528213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.528174 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:18:37.532761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.532740 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.551825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.551783 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-052254\"" Oct 30 07:18:37.552810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.552781 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:18:37.561245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.561210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:37.661932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.659286 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.661932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.659624 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tww9n\" (UniqueName: \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.769451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.769359 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tww9n\" (UniqueName: \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.769600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.769582 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.770570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.770542 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.795777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.795744 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tww9n\" (UniqueName: \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\") pod \"payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.848133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.848094 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:37.966697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.966604 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" event={"ID":"1209cabb-e68a-4402-89a2-3a8cc5f84a38","Type":"ContainerStarted","Data":"5816bdf3c810da2bee479adfc6e0bf9213676e2af6cb46db86822519c8d90d4b"} Oct 30 07:18:37.966697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:37.966684 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" event={"ID":"1209cabb-e68a-4402-89a2-3a8cc5f84a38","Type":"ContainerStarted","Data":"8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e"} Oct 30 07:18:38.019956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.015281 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" podStartSLOduration=2.01525807 podStartE2EDuration="2.01525807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:38.012962249 +0000 UTC m=+1281.703879688" watchObservedRunningTime="2026-10-30 07:18:38.01525807 +0000 UTC m=+1281.706175508" Oct 30 07:18:38.050405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.050365 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g"] Oct 30 07:18:38.056416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.056356 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-442b05d7-tx29g"] Oct 30 07:18:38.058433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.058383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:38.187261 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:18:38.187226 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74793825_cca2_4ace_82f4_94ae601225a9.slice/crio-f800d21ef38d677496e10a3da4c39cd9249e80929fc8c97e76d916fcd19956fb WatchSource:0}: Error finding container f800d21ef38d677496e10a3da4c39cd9249e80929fc8c97e76d916fcd19956fb: Status 404 returned error can't find the container with id f800d21ef38d677496e10a3da4c39cd9249e80929fc8c97e76d916fcd19956fb Oct 30 07:18:38.188078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.188049 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:18:38.199161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.199137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:38.434807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.434767 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b369d465-6781-465b-a0e2-ceb1b5aef310" path="/var/lib/kubelet/pods/b369d465-6781-465b-a0e2-ceb1b5aef310/volumes" Oct 30 07:18:38.500821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.500785 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:18:38.501055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.501028 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" containerID="cri-o://ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce" gracePeriod=30 Oct 30 07:18:38.511956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.511924 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:18:38.512145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.512123 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" containerID="cri-o://a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d" gracePeriod=30 Oct 30 07:18:38.555462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.555423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:38.981898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.981769 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" event={"ID":"74793825-cca2-4ace-82f4-94ae601225a9","Type":"ContainerStarted","Data":"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9"} Oct 30 07:18:38.982321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.982292 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" event={"ID":"74793825-cca2-4ace-82f4-94ae601225a9","Type":"ContainerStarted","Data":"f800d21ef38d677496e10a3da4c39cd9249e80929fc8c97e76d916fcd19956fb"} Oct 30 07:18:38.982436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.982348 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:38.984452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.984423 5894 generic.go:356] "Generic (PLEG): container finished" podID="1209cabb-e68a-4402-89a2-3a8cc5f84a38" containerID="5816bdf3c810da2bee479adfc6e0bf9213676e2af6cb46db86822519c8d90d4b" exitCode=0 Oct 30 07:18:38.984589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:38.984501 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" event={"ID":"1209cabb-e68a-4402-89a2-3a8cc5f84a38","Type":"ContainerDied","Data":"5816bdf3c810da2bee479adfc6e0bf9213676e2af6cb46db86822519c8d90d4b"} Oct 30 07:18:39.031304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:39.031245 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" podStartSLOduration=2.031230511 podStartE2EDuration="2.031230511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:39.028657708 +0000 UTC m=+1282.719575175" watchObservedRunningTime="2026-10-30 07:18:39.031230511 +0000 UTC m=+1282.722147981" Oct 30 07:18:39.979739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:39.979701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:40.198049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.198021 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:40.229601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.229565 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\" (UniqueName: \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\") pod \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " Oct 30 07:18:40.229851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.229685 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\") pod \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " Oct 30 07:18:40.229851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.229751 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\") pod \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\" (UID: \"1209cabb-e68a-4402-89a2-3a8cc5f84a38\") " Oct 30 07:18:40.230062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.230035 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca" pod "1209cabb-e68a-4402-89a2-3a8cc5f84a38" (UID: "1209cabb-e68a-4402-89a2-3a8cc5f84a38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:40.230382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.230363 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:40.231893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.231857 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca" pod "1209cabb-e68a-4402-89a2-3a8cc5f84a38" (UID: "1209cabb-e68a-4402-89a2-3a8cc5f84a38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:40.231893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.231861 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r" pod "1209cabb-e68a-4402-89a2-3a8cc5f84a38" (UID: "1209cabb-e68a-4402-89a2-3a8cc5f84a38"). InnerVolumeSpecName "kube-api-access-vpf9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:40.331534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.331494 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1209cabb-e68a-4402-89a2-3a8cc5f84a38-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:40.331534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:40.331529 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpf9r\" (UniqueName: \"kubernetes.io/projected/1209cabb-e68a-4402-89a2-3a8cc5f84a38-kube-api-access-vpf9r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:41.003219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:41.003178 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" Oct 30 07:18:41.003668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:41.003218 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z" event={"ID":"1209cabb-e68a-4402-89a2-3a8cc5f84a38","Type":"ContainerDied","Data":"8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e"} Oct 30 07:18:41.003668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:41.003297 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a1ddfb7d3774e9ae3986c7e45dbd89bd3d173da22e92bd4769ef8972578746e" Oct 30 07:18:41.644014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:41.643981 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:42.533956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.533919 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt"] Oct 30 07:18:42.534589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.534575 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1209cabb-e68a-4402-89a2-3a8cc5f84a38" containerName="revoke-keys" Oct 30 07:18:42.534683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.534592 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1209cabb-e68a-4402-89a2-3a8cc5f84a38" containerName="revoke-keys" Oct 30 07:18:42.534743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.534732 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1209cabb-e68a-4402-89a2-3a8cc5f84a38" containerName="revoke-keys" Oct 30 07:18:42.538468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.538449 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.568086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.568053 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:18:42.572987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.572957 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt"] Oct 30 07:18:42.580373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.580337 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:18:42.580525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.580459 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:18:42.580525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.580492 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:18:42.663756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.663718 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.663967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.663870 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ttv\" (UniqueName: \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.663967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.663946 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.765258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.765217 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.765462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.765336 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ttv\" (UniqueName: \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.765462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.765399 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.766018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.765997 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.768033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.768008 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.788934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.788872 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ttv\" (UniqueName: \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\") pod \"maas-api-revoke-sub-b2a2dbc4-6zdvt\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:42.854246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:42.854208 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:43.161817 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:18:43.161770 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413a5ac6_1ad2_41d0_8c1d_7ced23fc65e3.slice/crio-61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef WatchSource:0}: Error finding container 61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef: Status 404 returned error can't find the container with id 61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef Oct 30 07:18:43.162122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.162099 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt"] Oct 30 07:18:43.428919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.428828 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:43.821612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.821585 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:18:43.868609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.868582 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:18:43.882460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882209 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882377 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882421 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882473 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882529 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882576 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882580 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882610 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882683 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.882813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882749 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\") pod \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\" (UID: \"6b2f4984-85ca-4da3-9d66-9de09c972a8c\") " Oct 30 07:18:43.883200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.882940 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:43.883200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883156 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.883732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883419 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.883732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883441 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.883732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883453 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.884586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883830 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.884586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.883850 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.886920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.886887 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "kube-api-access-jvm2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.887030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.886982 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:43.887084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.887021 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.889207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.889181 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy" pod "6b2f4984-85ca-4da3-9d66-9de09c972a8c" (UID: "6b2f4984-85ca-4da3-9d66-9de09c972a8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.984206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984110 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984169 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984213 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984287 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984342 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984376 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984413 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984486 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984572 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\") pod \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\" (UID: \"c720d10e-e4b8-4ab3-9c6a-eec069012b74\") " Oct 30 07:18:43.984739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984685 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.984904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984811 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.984904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.984834 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:43.985050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985016 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.985117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985061 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.985485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985453 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985481 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985496 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985510 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985524 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985537 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985550 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985565 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985579 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985591 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b2f4984-85ca-4da3-9d66-9de09c972a8c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.985664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.985605 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvm2q\" (UniqueName: \"kubernetes.io/projected/6b2f4984-85ca-4da3-9d66-9de09c972a8c-kube-api-access-jvm2q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:43.986996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.986974 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "kube-api-access-7nt8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.987299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.987273 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:43.987299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.987292 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:18:43.987443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:43.987310 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo" pod "c720d10e-e4b8-4ab3-9c6a-eec069012b74" (UID: "c720d10e-e4b8-4ab3-9c6a-eec069012b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:18:44.028809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.028767 5894 generic.go:356] "Generic (PLEG): container finished" podID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerID="a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d" exitCode=0 Oct 30 07:18:44.029014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.028823 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" Oct 30 07:18:44.029014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.028899 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" event={"ID":"c720d10e-e4b8-4ab3-9c6a-eec069012b74","Type":"ContainerDied","Data":"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d"} Oct 30 07:18:44.029014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.028952 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss" event={"ID":"c720d10e-e4b8-4ab3-9c6a-eec069012b74","Type":"ContainerDied","Data":"277f85d23958bc3b9922e779d726a3494cf1bc8e73bdc95560ac3b4d730bc26b"} Oct 30 07:18:44.029014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.028993 5894 scope.go:122] "RemoveContainer" containerID="a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d" Oct 30 07:18:44.038559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.036863 5894 generic.go:356] "Generic (PLEG): container finished" podID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerID="ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce" exitCode=0 Oct 30 07:18:44.038559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.036936 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" event={"ID":"6b2f4984-85ca-4da3-9d66-9de09c972a8c","Type":"ContainerDied","Data":"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce"} Oct 30 07:18:44.038559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.036994 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" event={"ID":"6b2f4984-85ca-4da3-9d66-9de09c972a8c","Type":"ContainerDied","Data":"ac9caa636378dceda6c76a00d64abf29a48eff4b9ea54b5c0609ed2023dfeef0"} Oct 30 07:18:44.038559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.037121 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb" Oct 30 07:18:44.045512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.040581 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" event={"ID":"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3","Type":"ContainerStarted","Data":"632499a4bf5c4eb3c39750d68eab18aae9edae1c6967ea3774f08cb0983a35fe"} Oct 30 07:18:44.045512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.040682 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" event={"ID":"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3","Type":"ContainerStarted","Data":"61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef"} Oct 30 07:18:44.051876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.051855 5894 scope.go:122] "RemoveContainer" containerID="a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d" Oct 30 07:18:44.052485 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:44.052263 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d\": container with ID starting with a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d not found: ID does not exist" containerID="a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d" Oct 30 07:18:44.052485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.052293 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d"} err="failed to get container status \"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d\": rpc error: code = NotFound desc = could not find container \"a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d\": container with ID starting with a52ebf4c7d722038c9183c330058cc2e1ccdc3ce11a8e8148a53a7302b36256d not found: ID does not exist" Oct 30 07:18:44.052485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.052317 5894 scope.go:122] "RemoveContainer" containerID="ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce" Oct 30 07:18:44.065467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.065443 5894 scope.go:122] "RemoveContainer" containerID="ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce" Oct 30 07:18:44.065888 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:44.065860 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce\": container with ID starting with ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce not found: ID does not exist" containerID="ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce" Oct 30 07:18:44.065976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.065894 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce"} err="failed to get container status \"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce\": rpc error: code = NotFound desc = could not find container \"ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce\": container with ID starting with ae201a974cbe85dc26150829e79512573ff8063d08fb589e94c8313e304961ce not found: ID does not exist" Oct 30 07:18:44.086436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.086400 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:18:44.086611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.086534 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.086611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.086558 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.086611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.086571 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c720d10e-e4b8-4ab3-9c6a-eec069012b74-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.086611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.086583 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nt8p\" (UniqueName: \"kubernetes.io/projected/c720d10e-e4b8-4ab3-9c6a-eec069012b74-kube-api-access-7nt8p\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:44.102441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.102404 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-rzhss"] Oct 30 07:18:44.158872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.158806 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" podStartSLOduration=2.158783927 podStartE2EDuration="2.158783927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:18:44.155166477 +0000 UTC m=+1287.846083915" watchObservedRunningTime="2026-10-30 07:18:44.158783927 +0000 UTC m=+1287.849701365" Oct 30 07:18:44.195267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.195234 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:18:44.199102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.199072 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-56eb7f-openshift-default-69f79b8c5c-k6ngb"] Oct 30 07:18:44.439679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.439619 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" path="/var/lib/kubelet/pods/6b2f4984-85ca-4da3-9d66-9de09c972a8c/volumes" Oct 30 07:18:44.440427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:44.440405 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" path="/var/lib/kubelet/pods/c720d10e-e4b8-4ab3-9c6a-eec069012b74/volumes" Oct 30 07:18:45.049658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:45.049561 5894 generic.go:356] "Generic (PLEG): container finished" podID="413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" containerID="632499a4bf5c4eb3c39750d68eab18aae9edae1c6967ea3774f08cb0983a35fe" exitCode=0 Oct 30 07:18:45.050115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:45.049656 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" event={"ID":"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3","Type":"ContainerDied","Data":"632499a4bf5c4eb3c39750d68eab18aae9edae1c6967ea3774f08cb0983a35fe"} Oct 30 07:18:45.229781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:45.229736 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:46.031345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.031314 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz"] Oct 30 07:18:46.043797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.043761 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq"] Oct 30 07:18:46.046820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.046796 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a600f1a8-kxrgz"] Oct 30 07:18:46.049969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.049946 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449b92fb-98mfq"] Oct 30 07:18:46.230812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.230781 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:46.316705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.316580 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\" (UniqueName: \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\") pod \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " Oct 30 07:18:46.316889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.316743 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\") pod \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " Oct 30 07:18:46.316889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.316822 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\") pod \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\" (UID: \"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3\") " Oct 30 07:18:46.317511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.317475 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca" pod "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" (UID: "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:46.319870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.319841 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca" pod "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" (UID: "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:18:46.319870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.319868 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv" pod "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" (UID: "413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3"). InnerVolumeSpecName "kube-api-access-v8ttv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:46.419111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.419075 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ttv\" (UniqueName: \"kubernetes.io/projected/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-kube-api-access-v8ttv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:46.419111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.419102 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:46.419111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.419112 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:46.433035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.433006 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f768d86-8792-45f6-83d3-fb24034134c1" path="/var/lib/kubelet/pods/3f768d86-8792-45f6-83d3-fb24034134c1/volumes" Oct 30 07:18:46.433452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.433439 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9295b961-ff2f-47d6-a21f-094f58f7061e" path="/var/lib/kubelet/pods/9295b961-ff2f-47d6-a21f-094f58f7061e/volumes" Oct 30 07:18:46.946123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.946085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:46.994913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:46.994860 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:18:47.069150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.069115 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" Oct 30 07:18:47.069150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.069144 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt" event={"ID":"413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3","Type":"ContainerDied","Data":"61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef"} Oct 30 07:18:47.069756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.069223 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ffb2ef18804f8147a50dcbcc23f64d2947c9b519b3752919d4802e2669ebef" Oct 30 07:18:47.120790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.120752 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:18:47.343143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.343113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:47.464199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.464158 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:47.541954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:47.541916 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:18:48.519720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:48.517438 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:48.985383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:48.985352 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:49.110866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.110830 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:49.144265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.144225 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44"] Oct 30 07:18:49.156485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.156433 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-318524-wvq44"] Oct 30 07:18:49.225904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.225868 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:49.643060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.643023 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:49.996680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:49.996578 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:18:50.432427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:50.432393 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f81a86-fc31-4954-b156-6e8d2bd5fa37" path="/var/lib/kubelet/pods/32f81a86-fc31-4954-b156-6e8d2bd5fa37/volumes" Oct 30 07:18:50.902923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:50.902891 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:51.997303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:51.997223 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" podUID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" containerName="payload-processing" containerID="cri-o://7a25c2a0f72c8db4f8cfc33d9313d82e8452e504c147d271b980e2f670fb2f2a" gracePeriod=25 Oct 30 07:18:52.117349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.117305 5894 generic.go:356] "Generic (PLEG): container finished" podID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" containerID="7a25c2a0f72c8db4f8cfc33d9313d82e8452e504c147d271b980e2f670fb2f2a" exitCode=0 Oct 30 07:18:52.117465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.117371 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" event={"ID":"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a","Type":"ContainerDied","Data":"7a25c2a0f72c8db4f8cfc33d9313d82e8452e504c147d271b980e2f670fb2f2a"} Oct 30 07:18:52.121469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.121435 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" podUID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" containerName="payload-pre-processing" containerID="cri-o://618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b" gracePeriod=25 Oct 30 07:18:52.302906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.302877 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:18:52.420807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.420777 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\") pod \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " Oct 30 07:18:52.420957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.420837 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\" (UniqueName: \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\") pod \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\" (UID: \"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a\") " Oct 30 07:18:52.421275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.421160 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume" pod "1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" (UID: "1e5c646d-b7eb-4eec-a8e7-54e723b42a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:52.421498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.421482 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:52.423145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.423119 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn" pod "1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" (UID: "1e5c646d-b7eb-4eec-a8e7-54e723b42a4a"). InnerVolumeSpecName "kube-api-access-jtvgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:52.424943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.424925 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:18:52.522295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.522191 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\") pod \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " Oct 30 07:18:52.522478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.522326 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\" (UniqueName: \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\") pod \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\" (UID: \"ce6ed6e2-5ed7-4758-a337-0e2c6883859a\") " Oct 30 07:18:52.522655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.522610 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume" pod "ce6ed6e2-5ed7-4758-a337-0e2c6883859a" (UID: "ce6ed6e2-5ed7-4758-a337-0e2c6883859a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:18:52.523268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.523058 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvgn\" (UniqueName: \"kubernetes.io/projected/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a-kube-api-access-jtvgn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:52.523268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.523081 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:52.524517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.524495 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b" pod "ce6ed6e2-5ed7-4758-a337-0e2c6883859a" (UID: "ce6ed6e2-5ed7-4758-a337-0e2c6883859a"). InnerVolumeSpecName "kube-api-access-l8j6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:18:52.534443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.534405 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:52.624389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:52.624329 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8j6b\" (UniqueName: \"kubernetes.io/projected/ce6ed6e2-5ed7-4758-a337-0e2c6883859a-kube-api-access-l8j6b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:18:53.126569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.126539 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" Oct 30 07:18:53.127105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.126566 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh" event={"ID":"1e5c646d-b7eb-4eec-a8e7-54e723b42a4a","Type":"ContainerDied","Data":"59287821fb8ce0696d4ad489aba0a89837eb38db331d7d984dda3887e9e7a894"} Oct 30 07:18:53.127105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.126648 5894 scope.go:122] "RemoveContainer" containerID="7a25c2a0f72c8db4f8cfc33d9313d82e8452e504c147d271b980e2f670fb2f2a" Oct 30 07:18:53.128376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.128355 5894 generic.go:356] "Generic (PLEG): container finished" podID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" containerID="618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b" exitCode=0 Oct 30 07:18:53.128460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.128399 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" Oct 30 07:18:53.128460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.128443 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" event={"ID":"ce6ed6e2-5ed7-4758-a337-0e2c6883859a","Type":"ContainerDied","Data":"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b"} Oct 30 07:18:53.128570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.128510 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l" event={"ID":"ce6ed6e2-5ed7-4758-a337-0e2c6883859a","Type":"ContainerDied","Data":"bf1722ef8cd32d63267d56ef296dd7edde53d29455341fc3aaf06d45d6f90435"} Oct 30 07:18:53.138892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.138870 5894 scope.go:122] "RemoveContainer" containerID="618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b" Oct 30 07:18:53.148808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.148732 5894 scope.go:122] "RemoveContainer" containerID="618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b" Oct 30 07:18:53.149070 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:18:53.149040 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b\": container with ID starting with 618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b not found: ID does not exist" containerID="618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b" Oct 30 07:18:53.149164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.149074 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b"} err="failed to get container status \"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b\": rpc error: code = NotFound desc = could not find container \"618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b\": container with ID starting with 618f5e3619198c5ec2de93747ed7ed25ee28d19d139260f8ab3be4a9b43de81b not found: ID does not exist" Oct 30 07:18:53.183526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.183492 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:18:53.195791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.195761 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6cc5f2-7db5575b8-cqgmh"] Oct 30 07:18:53.224124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.224093 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:18:53.228732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.228703 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6cc5f2-6b676cfbdc-l6z9l"] Oct 30 07:18:53.813428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:53.813394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:54.203674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:54.203557 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:54.325314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:54.325283 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:54.450150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:54.450102 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:54.461373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:54.461295 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" path="/var/lib/kubelet/pods/1e5c646d-b7eb-4eec-a8e7-54e723b42a4a/volumes" Oct 30 07:18:54.462336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:54.462306 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" path="/var/lib/kubelet/pods/ce6ed6e2-5ed7-4758-a337-0e2c6883859a/volumes" Oct 30 07:18:56.115816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:56.115776 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:56.250676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:56.247507 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:56.597154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:56.597116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:56.812609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:56.812579 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:57.140926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.140888 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg"] Oct 30 07:18:57.141956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.141930 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" Oct 30 07:18:57.141956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.141954 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.141984 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" containerName="payload-pre-processing" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.141992 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" containerName="payload-pre-processing" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142027 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142035 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142062 5894 cpu_manager.go:414] "containerMap: removing container" podUID="413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" containerName="revoke-keys" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142070 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" containerName="revoke-keys" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142097 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" containerName="payload-processing" Oct 30 07:18:57.142152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142105 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" containerName="payload-processing" Oct 30 07:18:57.142432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142262 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c720d10e-e4b8-4ab3-9c6a-eec069012b74" containerName="istio-proxy" Oct 30 07:18:57.142432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142277 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" containerName="revoke-keys" Oct 30 07:18:57.142432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142291 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5c646d-b7eb-4eec-a8e7-54e723b42a4a" containerName="payload-processing" Oct 30 07:18:57.142432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142306 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2f4984-85ca-4da3-9d66-9de09c972a8c" containerName="istio-proxy" Oct 30 07:18:57.142432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.142320 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce6ed6e2-5ed7-4758-a337-0e2c6883859a" containerName="payload-pre-processing" Oct 30 07:18:57.146698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.146670 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.166991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.166960 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg"] Oct 30 07:18:57.173994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.173966 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:18:57.174801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.174776 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:18:57.176481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.175903 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:18:57.176621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.176011 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:18:57.305317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.305275 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.305529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.305435 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwtht\" (UniqueName: \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.305601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.305530 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.380916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.380877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:57.410680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.410050 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.415018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.411090 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwtht\" (UniqueName: \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.422937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.422903 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.431653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.424228 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.461662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.432712 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.462537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.462503 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwtht\" (UniqueName: \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\") pod \"maas-api-revoke-keys-e2e-api-b-466248-pgrbg\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.466899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.466869 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:18:57.790658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.790591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:57.806119 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:18:57.806084 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9eb8cd_699f_41f9_85ac_68543aeaa187.slice/crio-629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229 WatchSource:0}: Error finding container 629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229: Status 404 returned error can't find the container with id 629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229 Oct 30 07:18:57.806270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:57.806246 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg"] Oct 30 07:18:58.076276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:58.076196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:58.187446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:58.187411 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" event={"ID":"8d9eb8cd-699f-41f9-85ac-68543aeaa187","Type":"ContainerStarted","Data":"629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229"} Oct 30 07:18:58.205336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:58.205299 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:18:59.196922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:59.196841 5894 generic.go:356] "Generic (PLEG): container finished" podID="8d9eb8cd-699f-41f9-85ac-68543aeaa187" containerID="6dc4aa19b7b79c95528738e7c2fef07db4aa0cad62457fca9b1f5690890c34f5" exitCode=0 Oct 30 07:18:59.197292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:59.196932 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" event={"ID":"8d9eb8cd-699f-41f9-85ac-68543aeaa187","Type":"ContainerDied","Data":"6dc4aa19b7b79c95528738e7c2fef07db4aa0cad62457fca9b1f5690890c34f5"} Oct 30 07:18:59.820595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:18:59.820550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:00.106253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.106153 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:00.252082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.251987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:00.483066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.482989 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:19:00.619891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.619860 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\") pod \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " Oct 30 07:19:00.620063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.619945 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\") pod \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " Oct 30 07:19:00.620063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.619989 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\" (UniqueName: \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\") pod \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\" (UID: \"8d9eb8cd-699f-41f9-85ac-68543aeaa187\") " Oct 30 07:19:00.620327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.620301 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca" pod "8d9eb8cd-699f-41f9-85ac-68543aeaa187" (UID: "8d9eb8cd-699f-41f9-85ac-68543aeaa187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:00.622061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.622035 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca" pod "8d9eb8cd-699f-41f9-85ac-68543aeaa187" (UID: "8d9eb8cd-699f-41f9-85ac-68543aeaa187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:00.622133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.622098 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht" pod "8d9eb8cd-699f-41f9-85ac-68543aeaa187" (UID: "8d9eb8cd-699f-41f9-85ac-68543aeaa187"). InnerVolumeSpecName "kube-api-access-xwtht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:00.722093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.722008 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:00.722093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.722040 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d9eb8cd-699f-41f9-85ac-68543aeaa187-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:00.722093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:00.722049 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwtht\" (UniqueName: \"kubernetes.io/projected/8d9eb8cd-699f-41f9-85ac-68543aeaa187-kube-api-access-xwtht\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:01.223102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:01.223072 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" Oct 30 07:19:01.223294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:01.223098 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg" event={"ID":"8d9eb8cd-699f-41f9-85ac-68543aeaa187","Type":"ContainerDied","Data":"629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229"} Oct 30 07:19:01.223294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:01.223155 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="629bca4686ebdc0bed7a7f61e342a7a9a95b6e19909c78c5ac46c236a4803229" Oct 30 07:19:01.257035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:01.256988 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:01.389078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:01.389036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:02.113325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:02.113287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:02.914961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:02.914921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:03.030454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:03.030399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:03.831817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:03.831770 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:04.828575 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:04.828540 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:19:05.499744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:05.499702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:07.093498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:07.093465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:07.615969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:07.615933 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:19:07.648646 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:07.648589 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:19:07.753677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:07.753611 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:07.872049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:07.871963 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:08.714243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:08.714204 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:10.553377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:10.553343 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:12.271392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:12.271363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:12.616741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:12.616698 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" podUID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" containerName="payload-processing" containerID="cri-o://584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4" gracePeriod=25 Oct 30 07:19:12.649013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:12.648971 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" podUID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" containerName="payload-pre-processing" containerID="cri-o://574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168" gracePeriod=25 Oct 30 07:19:12.711015 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:12.710985 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6201fa9_d8c5_4eb8_a037_3d34c8391c7f.slice/crio-conmon-574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:19:12.980475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:12.980449 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:19:13.006918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.006891 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:19:13.041486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.041450 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\") pod \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " Oct 30 07:19:13.041953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.041919 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume" pod "c9eeaae2-60d5-47b8-811c-2bb6814035f3" (UID: "c9eeaae2-60d5-47b8-811c-2bb6814035f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:13.042111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.042042 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\" (UniqueName: \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\") pod \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\" (UID: \"c9eeaae2-60d5-47b8-811c-2bb6814035f3\") " Oct 30 07:19:13.042668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.042620 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9eeaae2-60d5-47b8-811c-2bb6814035f3-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:13.044783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.044754 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc" pod "c9eeaae2-60d5-47b8-811c-2bb6814035f3" (UID: "c9eeaae2-60d5-47b8-811c-2bb6814035f3"). InnerVolumeSpecName "kube-api-access-c29xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:13.143339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.143250 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\" (UniqueName: \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\") pod \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " Oct 30 07:19:13.143499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.143390 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\") pod \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\" (UID: \"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f\") " Oct 30 07:19:13.143789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.143764 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume" pod "e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" (UID: "e6201fa9-d8c5-4eb8-a037-3d34c8391c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:13.143968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.143952 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:13.144015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.143970 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c29xc\" (UniqueName: \"kubernetes.io/projected/c9eeaae2-60d5-47b8-811c-2bb6814035f3-kube-api-access-c29xc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:13.145646 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.145605 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn" pod "e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" (UID: "e6201fa9-d8c5-4eb8-a037-3d34c8391c7f"). InnerVolumeSpecName "kube-api-access-gv5dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:13.245513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.245474 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv5dn\" (UniqueName: \"kubernetes.io/projected/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f-kube-api-access-gv5dn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:13.438018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.437923 5894 generic.go:356] "Generic (PLEG): container finished" podID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" containerID="574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168" exitCode=0 Oct 30 07:19:13.438018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.437980 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" Oct 30 07:19:13.438551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.438011 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" event={"ID":"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f","Type":"ContainerDied","Data":"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168"} Oct 30 07:19:13.438551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.438070 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9" event={"ID":"e6201fa9-d8c5-4eb8-a037-3d34c8391c7f","Type":"ContainerDied","Data":"4c730fef951316255036ad46dcfec7d6e6e7288281c63188e8937332afd04420"} Oct 30 07:19:13.438551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.438095 5894 scope.go:122] "RemoveContainer" containerID="574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168" Oct 30 07:19:13.440449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.440417 5894 generic.go:356] "Generic (PLEG): container finished" podID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" containerID="584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4" exitCode=0 Oct 30 07:19:13.440614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.440545 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" Oct 30 07:19:13.440614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.440558 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" event={"ID":"c9eeaae2-60d5-47b8-811c-2bb6814035f3","Type":"ContainerDied","Data":"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4"} Oct 30 07:19:13.440614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.440607 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k" event={"ID":"c9eeaae2-60d5-47b8-811c-2bb6814035f3","Type":"ContainerDied","Data":"7632e08712fa4eaf74ac0db1232e234fcfa7b5353b9f702841bd81165d2aceb0"} Oct 30 07:19:13.451278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.451250 5894 scope.go:122] "RemoveContainer" containerID="574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168" Oct 30 07:19:13.451624 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:13.451599 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168\": container with ID starting with 574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168 not found: ID does not exist" containerID="574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168" Oct 30 07:19:13.451737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.451669 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168"} err="failed to get container status \"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168\": rpc error: code = NotFound desc = could not find container \"574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168\": container with ID starting with 574d80f667c591fdc46879a4206d5679bf2e4bdef20f5e68c72ee6c7990e6168 not found: ID does not exist" Oct 30 07:19:13.451737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.451701 5894 scope.go:122] "RemoveContainer" containerID="584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4" Oct 30 07:19:13.462144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.462127 5894 scope.go:122] "RemoveContainer" containerID="584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4" Oct 30 07:19:13.462483 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:13.462465 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4\": container with ID starting with 584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4 not found: ID does not exist" containerID="584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4" Oct 30 07:19:13.462523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.462493 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4"} err="failed to get container status \"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4\": rpc error: code = NotFound desc = could not find container \"584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4\": container with ID starting with 584ceb85c98ff0eea9691108dc59325b9833611ae90deee049e213a6acafd5e4 not found: ID does not exist" Oct 30 07:19:13.491206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.491172 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:19:13.504045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.504016 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-466248-6545d8dd96-rxj9k"] Oct 30 07:19:13.532298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.532264 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:19:13.538538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.538506 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-466248-5d4c4f94c9-js5k9"] Oct 30 07:19:13.945414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:13.945381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:14.432183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:14.432141 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" path="/var/lib/kubelet/pods/c9eeaae2-60d5-47b8-811c-2bb6814035f3/volumes" Oct 30 07:19:14.432661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:14.432629 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" path="/var/lib/kubelet/pods/e6201fa9-d8c5-4eb8-a037-3d34c8391c7f/volumes" Oct 30 07:19:15.828390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:15.828353 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:15.949839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:15.949797 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:16.063495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:16.063418 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:16.320163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:16.320128 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:16.962129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:16.962095 5894 scope.go:122] "RemoveContainer" containerID="9351320717c596a9ce1ee772f4a3e4d1df26da6d132b3f43683a509db75ea5be" Oct 30 07:19:16.981227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:16.981202 5894 scope.go:122] "RemoveContainer" containerID="d093e364d908c9b3b22feceaec34d15a80d5767f3180fa900906cdf050cda5ce" Oct 30 07:19:16.991553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:16.991522 5894 scope.go:122] "RemoveContainer" containerID="3c4aee608abce838a2d434ff5a3d5db8ffe68a78cc454442585e53dab0496817" Oct 30 07:19:17.003673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.003646 5894 scope.go:122] "RemoveContainer" containerID="f442045541c0c27d8cf5a564b0a35c27c4dba514b916a64dfe3aafefc9a9038d" Oct 30 07:19:17.014568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.014541 5894 scope.go:122] "RemoveContainer" containerID="d29cbce993a2c18ad76072b16f999770d21197858b5e2bc9d1b0081f73f61fdc" Oct 30 07:19:17.023835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.023815 5894 scope.go:122] "RemoveContainer" containerID="000ae1e872b7a3d84aeef9005d7707d127235ff238663be84096ee8ee56e2156" Oct 30 07:19:17.032907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.032886 5894 scope.go:122] "RemoveContainer" containerID="83727b111506ce56c22c6c330a14deb9a79f152930f21eb17781155a394fd703" Oct 30 07:19:17.043243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.043217 5894 scope.go:122] "RemoveContainer" containerID="38a7de25b3cbed389f42a7d6c449ce35a8e06266933c30746a6953abe7cd6b22" Oct 30 07:19:17.055713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.055691 5894 scope.go:122] "RemoveContainer" containerID="f3b13df7c29b4d7adc648f00cb8380db0ad3dec8c545419251b9a62d6068eb32" Oct 30 07:19:17.065724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.065703 5894 scope.go:122] "RemoveContainer" containerID="ec84a1b2f8e71c8b42008373f91aba1b9228a747cb8a7d27ec5acedff5a67684" Oct 30 07:19:17.075170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.075149 5894 scope.go:122] "RemoveContainer" containerID="0225b732fdb54b49936e117c5c781209c6068cec24e5de7f25f6b80975278f97" Oct 30 07:19:17.084113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.084092 5894 scope.go:122] "RemoveContainer" containerID="371c057669f661b724844a98c5db3768f9e42978c92613ad785c63235f515b18" Oct 30 07:19:17.220998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.220967 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q"] Oct 30 07:19:17.221821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221802 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" containerName="payload-processing" Oct 30 07:19:17.221821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221820 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" containerName="payload-processing" Oct 30 07:19:17.221978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221841 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8d9eb8cd-699f-41f9-85ac-68543aeaa187" containerName="revoke-keys" Oct 30 07:19:17.221978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221846 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9eb8cd-699f-41f9-85ac-68543aeaa187" containerName="revoke-keys" Oct 30 07:19:17.221978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221863 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" containerName="payload-pre-processing" Oct 30 07:19:17.221978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221869 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" containerName="payload-pre-processing" Oct 30 07:19:17.221978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221980 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9eeaae2-60d5-47b8-811c-2bb6814035f3" containerName="payload-processing" Oct 30 07:19:17.222164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.221994 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9eb8cd-699f-41f9-85ac-68543aeaa187" containerName="revoke-keys" Oct 30 07:19:17.222164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.222003 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6201fa9-d8c5-4eb8-a037-3d34c8391c7f" containerName="payload-pre-processing" Oct 30 07:19:17.229115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.229094 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.245922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.245891 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q"] Oct 30 07:19:17.260026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.254112 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:19:17.260026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.254225 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:19:17.287445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.287407 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:19:17.287609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.287460 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:19:17.309343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.309301 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.309581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.309401 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2crvr\" (UniqueName: \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.309581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.309491 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.411240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.411201 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.411446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.411312 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2crvr\" (UniqueName: \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.411446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.411425 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.412224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.412195 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.413898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.413868 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.435231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.435198 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2crvr\" (UniqueName: \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\") pod \"maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.545877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.545836 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:17.758829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.758786 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:17.857623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:17.856926 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q"] Oct 30 07:19:17.857623 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:19:17.857029 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ededa5c_e857_49b8_a58d_380692b10d9e.slice/crio-6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0 WatchSource:0}: Error finding container 6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0: Status 404 returned error can't find the container with id 6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0 Oct 30 07:19:18.488721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:18.488607 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" event={"ID":"8ededa5c-e857-49b8-a58d-380692b10d9e","Type":"ContainerStarted","Data":"1bd7354ecc4a04e62db2be662e192dde481924bec2f0591dd90c14bcbcf1b83c"} Oct 30 07:19:18.488721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:18.488691 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" event={"ID":"8ededa5c-e857-49b8-a58d-380692b10d9e","Type":"ContainerStarted","Data":"6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0"} Oct 30 07:19:18.534533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:18.534479 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" podStartSLOduration=1.5344646960000001 podStartE2EDuration="1.534464696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:19:18.531361064 +0000 UTC m=+1322.222278501" watchObservedRunningTime="2026-10-30 07:19:18.534464696 +0000 UTC m=+1322.225382132" Oct 30 07:19:19.498048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:19.498015 5894 generic.go:356] "Generic (PLEG): container finished" podID="8ededa5c-e857-49b8-a58d-380692b10d9e" containerID="1bd7354ecc4a04e62db2be662e192dde481924bec2f0591dd90c14bcbcf1b83c" exitCode=0 Oct 30 07:19:19.498440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:19.498093 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" event={"ID":"8ededa5c-e857-49b8-a58d-380692b10d9e","Type":"ContainerDied","Data":"1bd7354ecc4a04e62db2be662e192dde481924bec2f0591dd90c14bcbcf1b83c"} Oct 30 07:19:19.670360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:19.670330 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:19.781055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:19.780999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:19.893806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:19.893769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:20.690575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.690547 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:20.767244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.767156 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\") pod \"8ededa5c-e857-49b8-a58d-380692b10d9e\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " Oct 30 07:19:20.767408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.767345 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\" (UniqueName: \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\") pod \"8ededa5c-e857-49b8-a58d-380692b10d9e\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " Oct 30 07:19:20.767408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.767377 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\") pod \"8ededa5c-e857-49b8-a58d-380692b10d9e\" (UID: \"8ededa5c-e857-49b8-a58d-380692b10d9e\") " Oct 30 07:19:20.767617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.767580 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca" pod "8ededa5c-e857-49b8-a58d-380692b10d9e" (UID: "8ededa5c-e857-49b8-a58d-380692b10d9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:20.768452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.768419 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:20.772550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.772125 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca" pod "8ededa5c-e857-49b8-a58d-380692b10d9e" (UID: "8ededa5c-e857-49b8-a58d-380692b10d9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:20.785672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.785599 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr" pod "8ededa5c-e857-49b8-a58d-380692b10d9e" (UID: "8ededa5c-e857-49b8-a58d-380692b10d9e"). InnerVolumeSpecName "kube-api-access-2crvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:20.869798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.869759 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2crvr\" (UniqueName: \"kubernetes.io/projected/8ededa5c-e857-49b8-a58d-380692b10d9e-kube-api-access-2crvr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:20.869798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:20.869795 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ededa5c-e857-49b8-a58d-380692b10d9e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:21.456111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:21.456073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:21.518318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:21.518279 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" event={"ID":"8ededa5c-e857-49b8-a58d-380692b10d9e","Type":"ContainerDied","Data":"6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0"} Oct 30 07:19:21.518318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:21.518299 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q" Oct 30 07:19:21.518318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:21.518326 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6f38fd7f1bfa0e956e60ffe7fb377f6830caf052a503be411036b621b6aac0" Oct 30 07:19:23.070370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:23.070327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:24.680348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:24.680305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:26.343313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:26.343272 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:26.902474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:26.902439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:27.691395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:27.691356 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:19:27.722368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:27.722336 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:19:27.826391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:27.826337 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:28.027108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.027021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:28.432745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.432709 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:28.557346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.557310 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:28.576942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.576906 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z"] Oct 30 07:19:28.593702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.592565 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-6cc5f2-lfk7z"] Oct 30 07:19:28.676652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:28.676600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:29.107173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:29.107127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:30.328424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:30.328389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:30.433244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:30.433211 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1209cabb-e68a-4402-89a2-3a8cc5f84a38" path="/var/lib/kubelet/pods/1209cabb-e68a-4402-89a2-3a8cc5f84a38/volumes" Oct 30 07:19:32.030892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:32.030860 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:32.692439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:32.692392 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" podUID="222e7790-9c7a-4f88-a77d-96ce89ccd617" containerName="payload-processing" containerID="cri-o://a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c" gracePeriod=25 Oct 30 07:19:32.723246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:32.723211 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" podUID="5c2785b3-cc84-4e5a-92d3-115968622fb9" containerName="payload-pre-processing" containerID="cri-o://ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373" gracePeriod=25 Oct 30 07:19:32.807391 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:32.807359 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c2785b3_cc84_4e5a_92d3_115968622fb9.slice/crio-ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:19:32.998514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:32.998478 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:19:33.073286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.072088 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\" (UniqueName: \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\") pod \"222e7790-9c7a-4f88-a77d-96ce89ccd617\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " Oct 30 07:19:33.073286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.072142 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\") pod \"222e7790-9c7a-4f88-a77d-96ce89ccd617\" (UID: \"222e7790-9c7a-4f88-a77d-96ce89ccd617\") " Oct 30 07:19:33.073286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.073243 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume" pod "222e7790-9c7a-4f88-a77d-96ce89ccd617" (UID: "222e7790-9c7a-4f88-a77d-96ce89ccd617"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:33.075194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.075168 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft" pod "222e7790-9c7a-4f88-a77d-96ce89ccd617" (UID: "222e7790-9c7a-4f88-a77d-96ce89ccd617"). InnerVolumeSpecName "kube-api-access-6dlft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:33.091229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.091201 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:19:33.173308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.173267 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\") pod \"5c2785b3-cc84-4e5a-92d3-115968622fb9\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " Oct 30 07:19:33.173511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.173392 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\" (UniqueName: \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\") pod \"5c2785b3-cc84-4e5a-92d3-115968622fb9\" (UID: \"5c2785b3-cc84-4e5a-92d3-115968622fb9\") " Oct 30 07:19:33.173618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.173595 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume" pod "5c2785b3-cc84-4e5a-92d3-115968622fb9" (UID: "5c2785b3-cc84-4e5a-92d3-115968622fb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:33.174105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.174084 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c2785b3-cc84-4e5a-92d3-115968622fb9-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:33.174105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.174106 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dlft\" (UniqueName: \"kubernetes.io/projected/222e7790-9c7a-4f88-a77d-96ce89ccd617-kube-api-access-6dlft\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:33.174264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.174116 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/222e7790-9c7a-4f88-a77d-96ce89ccd617-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:33.175862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.175833 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx" pod "5c2785b3-cc84-4e5a-92d3-115968622fb9" (UID: "5c2785b3-cc84-4e5a-92d3-115968622fb9"). InnerVolumeSpecName "kube-api-access-5nsqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:33.275563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.275488 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nsqx\" (UniqueName: \"kubernetes.io/projected/5c2785b3-cc84-4e5a-92d3-115968622fb9-kube-api-access-5nsqx\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:33.614205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.614170 5894 generic.go:356] "Generic (PLEG): container finished" podID="222e7790-9c7a-4f88-a77d-96ce89ccd617" containerID="a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c" exitCode=0 Oct 30 07:19:33.614421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.614228 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" Oct 30 07:19:33.614421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.614256 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" event={"ID":"222e7790-9c7a-4f88-a77d-96ce89ccd617","Type":"ContainerDied","Data":"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c"} Oct 30 07:19:33.614421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.614305 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2" event={"ID":"222e7790-9c7a-4f88-a77d-96ce89ccd617","Type":"ContainerDied","Data":"d93c9e92bd33b8f11e4accd21bcf1ebac8a20156e51b9e34417d779754a22790"} Oct 30 07:19:33.614421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.614328 5894 scope.go:122] "RemoveContainer" containerID="a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c" Oct 30 07:19:33.615983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.615958 5894 generic.go:356] "Generic (PLEG): container finished" podID="5c2785b3-cc84-4e5a-92d3-115968622fb9" containerID="ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373" exitCode=0 Oct 30 07:19:33.616103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.615988 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" event={"ID":"5c2785b3-cc84-4e5a-92d3-115968622fb9","Type":"ContainerDied","Data":"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373"} Oct 30 07:19:33.616103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.616021 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" Oct 30 07:19:33.616209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.616030 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr" event={"ID":"5c2785b3-cc84-4e5a-92d3-115968622fb9","Type":"ContainerDied","Data":"c99dc79df3f4fb994aaf90146cab53acf7893bd8d04c5cb17b543c863294383b"} Oct 30 07:19:33.626944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.626923 5894 scope.go:122] "RemoveContainer" containerID="a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c" Oct 30 07:19:33.627240 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:33.627219 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c\": container with ID starting with a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c not found: ID does not exist" containerID="a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c" Oct 30 07:19:33.627297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.627251 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c"} err="failed to get container status \"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c\": rpc error: code = NotFound desc = could not find container \"a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c\": container with ID starting with a0cbb8901bda4fef4d59084152a66b98298a23719b1c623eda94e15de725811c not found: ID does not exist" Oct 30 07:19:33.627297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.627270 5894 scope.go:122] "RemoveContainer" containerID="ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373" Oct 30 07:19:33.637307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.637289 5894 scope.go:122] "RemoveContainer" containerID="ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373" Oct 30 07:19:33.637591 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:33.637573 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373\": container with ID starting with ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373 not found: ID does not exist" containerID="ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373" Oct 30 07:19:33.637648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.637600 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373"} err="failed to get container status \"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373\": rpc error: code = NotFound desc = could not find container \"ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373\": container with ID starting with ac10d43c813b4e6550ef816b65d3daaacea6f53c01c9e051409e31d7bad97373 not found: ID does not exist" Oct 30 07:19:33.667566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.667526 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:19:33.674015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.673979 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4c43f7-79879c5f88-r6gd2"] Oct 30 07:19:33.710700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.710665 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:19:33.712505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.712477 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:33.715460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:33.715438 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4c43f7-dc69657f5-pqfqr"] Oct 30 07:19:34.131825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:34.131674 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:19:34.431875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:34.431789 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222e7790-9c7a-4f88-a77d-96ce89ccd617" path="/var/lib/kubelet/pods/222e7790-9c7a-4f88-a77d-96ce89ccd617/volumes" Oct 30 07:19:34.432216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:34.432202 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c2785b3-cc84-4e5a-92d3-115968622fb9" path="/var/lib/kubelet/pods/5c2785b3-cc84-4e5a-92d3-115968622fb9/volumes" Oct 30 07:19:34.827747 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:19:34.827710 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:19:35.144516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:35.144430 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:35.519063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:35.518943 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:35.956990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:35.956957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:36.161171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.161134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:36.321700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.321656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:36.926464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.926425 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7"] Oct 30 07:19:36.927096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927073 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8ededa5c-e857-49b8-a58d-380692b10d9e" containerName="revoke-keys" Oct 30 07:19:36.927096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927092 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ededa5c-e857-49b8-a58d-380692b10d9e" containerName="revoke-keys" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927130 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5c2785b3-cc84-4e5a-92d3-115968622fb9" containerName="payload-pre-processing" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927136 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c2785b3-cc84-4e5a-92d3-115968622fb9" containerName="payload-pre-processing" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927153 5894 cpu_manager.go:414] "containerMap: removing container" podUID="222e7790-9c7a-4f88-a77d-96ce89ccd617" containerName="payload-processing" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927158 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222e7790-9c7a-4f88-a77d-96ce89ccd617" containerName="payload-processing" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927258 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ededa5c-e857-49b8-a58d-380692b10d9e" containerName="revoke-keys" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927281 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c2785b3-cc84-4e5a-92d3-115968622fb9" containerName="payload-pre-processing" Oct 30 07:19:36.927309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.927293 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="222e7790-9c7a-4f88-a77d-96ce89ccd617" containerName="payload-processing" Oct 30 07:19:36.931049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.931024 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:36.991409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:36.991377 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7"] Oct 30 07:19:37.000671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.000628 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:19:37.001025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.000971 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:19:37.001025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.000976 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:19:37.045058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.045010 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.045274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.045230 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzzp\" (UniqueName: \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.045405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.045390 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.067383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.067349 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:19:37.100089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.100057 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g"] Oct 30 07:19:37.116865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.116830 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1d30bd-4zj8g"] Oct 30 07:19:37.146501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.146468 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.146776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.146613 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzzp\" (UniqueName: \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.146776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.146766 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.147337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.147311 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.149446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.149424 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.171439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.171413 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzzp\" (UniqueName: \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\") pod \"maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.248927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.248815 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:37.600007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.599972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:37.663135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:37.663092 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7"] Oct 30 07:19:37.663335 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:19:37.663311 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f849b00_a0c0_44f6_9104_4cb3c9a7ae5f.slice/crio-ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5 WatchSource:0}: Error finding container ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5: Status 404 returned error can't find the container with id ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5 Oct 30 07:19:38.439052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:38.439010 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7e772f9-71a0-4de2-9c4b-a549fc649a60" path="/var/lib/kubelet/pods/b7e772f9-71a0-4de2-9c4b-a549fc649a60/volumes" Oct 30 07:19:38.667604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:38.667574 5894 generic.go:356] "Generic (PLEG): container finished" podID="1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" containerID="fa14cae109660e5caee3d2ebd6ec6b0177a4de16c09b3fb6962ba78c2bbe4223" exitCode=0 Oct 30 07:19:38.667837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:38.667680 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" event={"ID":"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f","Type":"ContainerDied","Data":"fa14cae109660e5caee3d2ebd6ec6b0177a4de16c09b3fb6962ba78c2bbe4223"} Oct 30 07:19:38.667837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:38.667742 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" event={"ID":"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f","Type":"ContainerStarted","Data":"ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5"} Oct 30 07:19:39.106949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:39.106910 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:39.225804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:39.225763 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:39.345975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:39.343240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:39.515058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:39.514968 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:39.902344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:39.902311 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:40.028797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.028758 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\") pod \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " Oct 30 07:19:40.029011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.028998 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\") pod \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " Oct 30 07:19:40.029061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.029041 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\" (UniqueName: \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\") pod \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\" (UID: \"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f\") " Oct 30 07:19:40.029333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.029306 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca" pod "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" (UID: "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:19:40.029556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.029535 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.031037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.031012 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca" pod "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" (UID: "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:19:40.031185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.031163 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp" pod "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" (UID: "1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f"). InnerVolumeSpecName "kube-api-access-hhzzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:19:40.131308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.131227 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzzp\" (UniqueName: \"kubernetes.io/projected/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-kube-api-access-hhzzp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.131308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.131258 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:19:40.683545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.683516 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" Oct 30 07:19:40.683988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.683552 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7" event={"ID":"1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f","Type":"ContainerDied","Data":"ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5"} Oct 30 07:19:40.683988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:40.683597 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea308f07c4fead54ae8ba44984996c45b4a51ed874f5d3c66a21a6e00d2accb5" Oct 30 07:19:41.212441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:41.212406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:42.089152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:42.089118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:42.203398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:42.203358 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:43.071499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:43.071463 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:44.573077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:44.573037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:45.140810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.140777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:45.257624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.257580 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:45.280082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.280053 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg"] Oct 30 07:19:45.292312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.292280 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-466248-pgrbg"] Oct 30 07:19:45.372296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.372253 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:45.780265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:45.780227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:46.435264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:46.435224 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9eb8cd-699f-41f9-85ac-68543aeaa187" path="/var/lib/kubelet/pods/8d9eb8cd-699f-41f9-85ac-68543aeaa187/volumes" Oct 30 07:19:47.096762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:47.096714 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:47.637757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:47.637720 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:47.754552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:47.754516 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:48.856109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:48.856062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:50.478626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:50.478579 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:51.926458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:51.926419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:52.058172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:52.058141 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:52.251014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:52.250930 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:52.385770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:52.385733 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:53.550691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:53.550622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:54.050694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:54.050618 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:54.203219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:54.203178 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:54.383394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:54.383281 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:55.355998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:55.355961 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:55.524193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:55.524137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:56.533607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:56.533574 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:58.615507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:58.615472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:58.738039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:58.738006 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:58.868832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:58.868745 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:19:59.999468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:19:59.999422 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:00.177132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:00.177072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:00.584033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:00.583999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:00.711983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:00.711938 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:01.259451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:01.259411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:02.921739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:02.921701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:04.322084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:04.322046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:04.832731 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:20:04.832695 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:20:04.914530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:04.914496 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:05.014352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:05.014319 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q"] Oct 30 07:20:05.026650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:05.026590 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4c43f7-x9p4q"] Oct 30 07:20:05.034870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:05.034834 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:05.517873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:05.517842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:06.431289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:06.431256 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ededa5c-e857-49b8-a58d-380692b10d9e" path="/var/lib/kubelet/pods/8ededa5c-e857-49b8-a58d-380692b10d9e/volumes" Oct 30 07:20:06.849701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:06.849669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:08.184753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:08.184717 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:20:08.661239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:08.661193 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:10.347049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:10.347012 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:11.156548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:11.156500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:11.637536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:11.637499 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:11.779223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:11.779185 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:11.901342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:11.901248 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:12.023167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.023132 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:12.144796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.144762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:12.265714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.263429 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:12.701511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.701481 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l"] Oct 30 07:20:12.702128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.702112 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" containerName="revoke-keys" Oct 30 07:20:12.702179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.702128 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" containerName="revoke-keys" Oct 30 07:20:12.702258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.702249 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" containerName="revoke-keys" Oct 30 07:20:12.706465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.706447 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.725967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.725937 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l"] Oct 30 07:20:12.734481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.734446 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:20:12.734481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.734456 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:20:12.734792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.734595 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:20:12.737219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.737192 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:20:12.803090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.803054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.803262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.803202 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwwb\" (UniqueName: \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.803320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.803273 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.904299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.904262 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwwb\" (UniqueName: \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.904498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.904326 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.904565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.904526 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.905013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.904992 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.906986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.906960 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:12.927859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:12.927825 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwwb\" (UniqueName: \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\") pod \"maas-api-revoke-keys-e2e-api-b-052254-s596l\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:13.020661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:13.020564 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:13.350696 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:20:13.350661 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9decb650_5114_42e4_934c_36ea5d4b98ff.slice/crio-48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5 WatchSource:0}: Error finding container 48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5: Status 404 returned error can't find the container with id 48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5 Oct 30 07:20:13.350949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:13.350925 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l"] Oct 30 07:20:14.029379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:14.029341 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" event={"ID":"9decb650-5114-42e4-934c-36ea5d4b98ff","Type":"ContainerStarted","Data":"c4bb07b42fb6beeab53dc585a190443a58ceb6e9ba6f61743e62b39ab93eff9c"} Oct 30 07:20:14.029941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:14.029406 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" event={"ID":"9decb650-5114-42e4-934c-36ea5d4b98ff","Type":"ContainerStarted","Data":"48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5"} Oct 30 07:20:14.070051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:14.069992 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" podStartSLOduration=2.069977598 podStartE2EDuration="2.069977598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:20:14.069403686 +0000 UTC m=+1377.760321124" watchObservedRunningTime="2026-10-30 07:20:14.069977598 +0000 UTC m=+1377.760895034" Oct 30 07:20:14.557530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:14.557453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:15.063656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:15.042887 5894 generic.go:356] "Generic (PLEG): container finished" podID="9decb650-5114-42e4-934c-36ea5d4b98ff" containerID="c4bb07b42fb6beeab53dc585a190443a58ceb6e9ba6f61743e62b39ab93eff9c" exitCode=0 Oct 30 07:20:15.063656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:15.043185 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" event={"ID":"9decb650-5114-42e4-934c-36ea5d4b98ff","Type":"ContainerDied","Data":"c4bb07b42fb6beeab53dc585a190443a58ceb6e9ba6f61743e62b39ab93eff9c"} Oct 30 07:20:16.223325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.223295 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:16.360833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.360734 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\") pod \"9decb650-5114-42e4-934c-36ea5d4b98ff\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " Oct 30 07:20:16.361021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.360900 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\" (UniqueName: \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\") pod \"9decb650-5114-42e4-934c-36ea5d4b98ff\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " Oct 30 07:20:16.361021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.360937 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\") pod \"9decb650-5114-42e4-934c-36ea5d4b98ff\" (UID: \"9decb650-5114-42e4-934c-36ea5d4b98ff\") " Oct 30 07:20:16.361987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.361952 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca" pod "9decb650-5114-42e4-934c-36ea5d4b98ff" (UID: "9decb650-5114-42e4-934c-36ea5d4b98ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:16.363687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.363616 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca" pod "9decb650-5114-42e4-934c-36ea5d4b98ff" (UID: "9decb650-5114-42e4-934c-36ea5d4b98ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:20:16.363992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.363962 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb" pod "9decb650-5114-42e4-934c-36ea5d4b98ff" (UID: "9decb650-5114-42e4-934c-36ea5d4b98ff"). InnerVolumeSpecName "kube-api-access-qhwwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:16.462534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.462507 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:20:16.462534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.462532 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwwb\" (UniqueName: \"kubernetes.io/projected/9decb650-5114-42e4-934c-36ea5d4b98ff-kube-api-access-qhwwb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:20:16.462762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:16.462543 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9decb650-5114-42e4-934c-36ea5d4b98ff-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:20:17.058902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.058870 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" Oct 30 07:20:17.059121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.058860 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l" event={"ID":"9decb650-5114-42e4-934c-36ea5d4b98ff","Type":"ContainerDied","Data":"48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5"} Oct 30 07:20:17.059121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.059026 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48fba66b8605690eb70d2af5da2fd6a49ee2d888204d8ab7c19ecf03f2ceacb5" Oct 30 07:20:17.227942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.227914 5894 scope.go:122] "RemoveContainer" containerID="8f79ac2f6519e7b4bb400a74dbb8a09c50045ed3082617a084e2769b30c6f40b" Oct 30 07:20:17.238260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.238238 5894 scope.go:122] "RemoveContainer" containerID="6f40e6ad24ef90a6cab6448ab130fab86a69e90f30a029ac141c7e81e3e7d9a3" Oct 30 07:20:17.247668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.247648 5894 scope.go:122] "RemoveContainer" containerID="950d76af2c628cc2a8f7cfc8b9da38c8f35b96b90e40ca2d238fea9ae6e8741e" Oct 30 07:20:17.257433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.257412 5894 scope.go:122] "RemoveContainer" containerID="10a5a01d3a5dfed2d2191383822be943160b8498875a9c641ea4ebc677faba15" Oct 30 07:20:17.266696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.266676 5894 scope.go:122] "RemoveContainer" containerID="c672780c90c7273013d8c734e4828f86a28cd78c5bb6bbfe46263a284c495cca" Oct 30 07:20:17.324249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.324217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:17.470083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:17.470050 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:18.021930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:18.021902 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:18.147344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:18.147305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:18.782612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:18.782575 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:19.248841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:19.248747 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:19.751539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:19.751486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:20.076440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:20.076406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:21.243678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:21.240073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:22.920828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:22.920794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:23.199345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:23.199263 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:20:23.356326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:23.356294 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:23.479211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:23.479127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:23.601014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:23.600980 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:24.871714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:24.871679 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:25.238436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:25.238358 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:25.411803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:25.411766 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:26.032310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:26.032274 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9"] Oct 30 07:20:26.044683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:26.044652 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff4d78c4-cxpt9"] Oct 30 07:20:26.393262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:26.393177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:26.440672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:26.440610 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82661356-13e8-4a6e-ae5e-78b399bfd05a" path="/var/lib/kubelet/pods/82661356-13e8-4a6e-ae5e-78b399bfd05a/volumes" Oct 30 07:20:26.593436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:26.593394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:27.020159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:27.020130 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:27.030473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:27.030439 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697"] Oct 30 07:20:27.042887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:27.042856 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02b49869-q5697"] Oct 30 07:20:28.199730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.199692 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" podUID="74793825-cca2-4ace-82f4-94ae601225a9" containerName="payload-processing" containerID="cri-o://aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9" gracePeriod=25 Oct 30 07:20:28.317207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.317174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:28.444758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.444710 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281e7884-05ef-49c5-a1a1-33a3bf42eda1" path="/var/lib/kubelet/pods/281e7884-05ef-49c5-a1a1-33a3bf42eda1/volumes" Oct 30 07:20:28.502147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.502123 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:20:28.582227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.582192 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\") pod \"74793825-cca2-4ace-82f4-94ae601225a9\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " Oct 30 07:20:28.582429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.582260 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\" (UniqueName: \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\") pod \"74793825-cca2-4ace-82f4-94ae601225a9\" (UID: \"74793825-cca2-4ace-82f4-94ae601225a9\") " Oct 30 07:20:28.582699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.582629 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume" pod "74793825-cca2-4ace-82f4-94ae601225a9" (UID: "74793825-cca2-4ace-82f4-94ae601225a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:20:28.582919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.582898 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74793825-cca2-4ace-82f4-94ae601225a9-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:20:28.584585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.584560 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n" pod "74793825-cca2-4ace-82f4-94ae601225a9" (UID: "74793825-cca2-4ace-82f4-94ae601225a9"). InnerVolumeSpecName "kube-api-access-tww9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:20:28.684421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:28.684387 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tww9n\" (UniqueName: \"kubernetes.io/projected/74793825-cca2-4ace-82f4-94ae601225a9-kube-api-access-tww9n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:20:29.160171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.160127 5894 generic.go:356] "Generic (PLEG): container finished" podID="74793825-cca2-4ace-82f4-94ae601225a9" containerID="aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9" exitCode=0 Oct 30 07:20:29.160171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.160177 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" Oct 30 07:20:29.160405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.160170 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" event={"ID":"74793825-cca2-4ace-82f4-94ae601225a9","Type":"ContainerDied","Data":"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9"} Oct 30 07:20:29.160405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.160312 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8" event={"ID":"74793825-cca2-4ace-82f4-94ae601225a9","Type":"ContainerDied","Data":"f800d21ef38d677496e10a3da4c39cd9249e80929fc8c97e76d916fcd19956fb"} Oct 30 07:20:29.160405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.160344 5894 scope.go:122] "RemoveContainer" containerID="aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9" Oct 30 07:20:29.171913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.171893 5894 scope.go:122] "RemoveContainer" containerID="aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9" Oct 30 07:20:29.172220 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:20:29.172200 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9\": container with ID starting with aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9 not found: ID does not exist" containerID="aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9" Oct 30 07:20:29.172264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.172230 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9"} err="failed to get container status \"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9\": rpc error: code = NotFound desc = could not find container \"aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9\": container with ID starting with aadc0e272ac1fe424348fab9ed08ec023df438c091a60168439a968be6c40af9 not found: ID does not exist" Oct 30 07:20:29.212746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.212707 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:20:29.225168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.225137 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-052254-7d4f4b685b-7rmj8"] Oct 30 07:20:29.724294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:29.724260 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:30.219926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.219889 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:30.334361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.334329 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:30.390443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.390404 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7"] Oct 30 07:20:30.402712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.402673 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-486cd4-4hcx7"] Oct 30 07:20:30.432917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.432883 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f" path="/var/lib/kubelet/pods/1f849b00-a0c0-44f6-9104-4cb3c9a7ae5f/volumes" Oct 30 07:20:30.433400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.433383 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74793825-cca2-4ace-82f4-94ae601225a9" path="/var/lib/kubelet/pods/74793825-cca2-4ace-82f4-94ae601225a9/volumes" Oct 30 07:20:30.466164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:30.463757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:31.026924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:31.026873 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:31.035505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:31.035472 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b"] Oct 30 07:20:31.047841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:31.047809 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4958a592-d8v9b"] Oct 30 07:20:32.432021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:32.431985 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bdc39e2-6892-491b-96d2-353a0f8ef512" path="/var/lib/kubelet/pods/0bdc39e2-6892-491b-96d2-353a0f8ef512/volumes" Oct 30 07:20:33.151820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:33.151766 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:34.828703 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:20:34.828665 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:20:34.999344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:34.999310 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:35.119617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:35.119518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:35.232522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:35.232486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:36.917123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:36.917087 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:37.975527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:37.975497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:38.097202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:38.097168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:38.286964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:38.286917 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:39.027106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:39.027073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:39.231511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:39.231469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:39.912327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:39.912288 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:40.050725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:40.042083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:41.668670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:41.668606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:43.386382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:43.386346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:43.514099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:43.510545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:43.622844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:43.622811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:44.055417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:44.055376 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:44.371872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:44.371782 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:44.703406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:44.703325 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:44.820553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:44.820516 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:45.647509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:45.647477 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:47.116394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:47.116353 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:47.264459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:47.264423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:47.606030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:47.605954 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:49.334833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:49.334801 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:50.889513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:50.889480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:51.407861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:51.407745 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:51.542251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:51.542205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:51.572026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:51.571967 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l"] Oct 30 07:20:51.585380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:51.585351 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-052254-s596l"] Oct 30 07:20:51.669784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:51.669693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:52.075165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:52.075104 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:52.431923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:52.431845 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9decb650-5114-42e4-934c-36ea5d4b98ff" path="/var/lib/kubelet/pods/9decb650-5114-42e4-934c-36ea5d4b98ff/volumes" Oct 30 07:20:53.368566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:53.368533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:55.526837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:55.526796 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:57.156654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:57.156275 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:20:57.353004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:57.352965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:59.174572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:59.174535 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:59.284422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:59.284380 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:59.414192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:59.414155 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:20:59.543672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:20:59.543610 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:00.520099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:00.520060 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:00.719075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:00.719044 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:00.930742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:00.926103 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:01.188522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:01.188433 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:01.366116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:01.366083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:02.029975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:02.029944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:02.167821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:02.167789 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:02.287924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:02.287825 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:04.229820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:04.229784 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:04.828900 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:21:04.828867 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:21:05.904307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:05.904273 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:06.093976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:06.093942 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:06.222498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:06.222401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:07.939328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:07.939290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:08.059574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:08.059536 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:09.707875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:09.707836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:11.521890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:11.521853 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:13.127529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:13.127498 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:14.677805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:14.677767 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:14.813730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:14.813689 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:14.930411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:14.930255 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:16.539929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:16.539892 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:16.747026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:16.746990 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:17.275903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:17.275864 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:17.340660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:17.331700 5894 scope.go:122] "RemoveContainer" containerID="f59cc9570588f753c57735f9b3508bb8269b9d82bec68b2abedc2efbe17dbd0a" Oct 30 07:21:17.495090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:17.494417 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:18.991124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:18.991088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:20.670316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:20.670280 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:21.893525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:21.893490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:22.090152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:22.090118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:22.597473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:22.597435 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:22.726603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:22.726563 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:23.227117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:23.227076 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:24.029182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:24.029146 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq"] Oct 30 07:21:24.041323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:24.041291 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac1b07b6-28vgq"] Oct 30 07:21:24.407435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:24.407354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:24.433066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:24.433028 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f80afe2-da7f-4d2f-85dc-1900abd04a1e" path="/var/lib/kubelet/pods/7f80afe2-da7f-4d2f-85dc-1900abd04a1e/volumes" Oct 30 07:21:24.517506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:24.517474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:25.764257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:25.764221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:27.141721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:27.141623 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:27.296751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:27.296712 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:21:27.950395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:27.950365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:29.580405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:29.580371 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:30.919095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:30.919056 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:31.389806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:31.389762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:33.144250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:33.144220 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:34.465924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:34.465891 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:34.643320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:34.643269 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:34.835480 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:21:34.835426 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:21:35.437968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:35.437931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:35.563783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:35.563749 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:35.688902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:35.688811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:37.322903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:37.322863 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:39.071829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:39.071794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:40.367051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367022 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc"] Oct 30 07:21:40.367614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367599 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9decb650-5114-42e4-934c-36ea5d4b98ff" containerName="revoke-keys" Oct 30 07:21:40.367696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367615 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9decb650-5114-42e4-934c-36ea5d4b98ff" containerName="revoke-keys" Oct 30 07:21:40.367696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367682 5894 cpu_manager.go:414] "containerMap: removing container" podUID="74793825-cca2-4ace-82f4-94ae601225a9" containerName="payload-processing" Oct 30 07:21:40.367696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367687 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74793825-cca2-4ace-82f4-94ae601225a9" containerName="payload-processing" Oct 30 07:21:40.367798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367770 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="74793825-cca2-4ace-82f4-94ae601225a9" containerName="payload-processing" Oct 30 07:21:40.367798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.367779 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9decb650-5114-42e4-934c-36ea5d4b98ff" containerName="revoke-keys" Oct 30 07:21:40.371239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.371209 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.399253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.399219 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.399428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.399371 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvn2h\" (UniqueName: \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.399428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.399417 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.429735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.429700 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:21:40.429928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.429835 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:21:40.433839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.433811 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc"] Oct 30 07:21:40.491882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.491848 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:21:40.492061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.491946 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:21:40.500591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.500553 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.500782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.500754 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvn2h\" (UniqueName: \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.500855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.500823 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.501424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.501396 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.503156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.503135 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.527535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.527505 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvn2h\" (UniqueName: \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\") pod \"maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.685547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.685461 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:40.939250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:40.939119 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:41.004111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:41.004052 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc"] Oct 30 07:21:41.007025 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:21:41.006994 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df2c15a_2f4f_4f6b_b29f_18d0a3898961.slice/crio-55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d WatchSource:0}: Error finding container 55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d: Status 404 returned error can't find the container with id 55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d Oct 30 07:21:41.899261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:41.899233 5894 generic.go:356] "Generic (PLEG): container finished" podID="9df2c15a-2f4f-4f6b-b29f-18d0a3898961" containerID="e7d34020f7988d8357f90bb583baf65caa7126369656c7d5c747fa73c946485a" exitCode=0 Oct 30 07:21:41.899649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:41.899310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" event={"ID":"9df2c15a-2f4f-4f6b-b29f-18d0a3898961","Type":"ContainerDied","Data":"e7d34020f7988d8357f90bb583baf65caa7126369656c7d5c747fa73c946485a"} Oct 30 07:21:41.899649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:41.899367 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" event={"ID":"9df2c15a-2f4f-4f6b-b29f-18d0a3898961","Type":"ContainerStarted","Data":"55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d"} Oct 30 07:21:42.749084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:42.749052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:43.076150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.076121 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:43.137073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.137035 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\") pod \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " Oct 30 07:21:43.137227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.137089 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\" (UniqueName: \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\") pod \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " Oct 30 07:21:43.137307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.137291 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\") pod \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\" (UID: \"9df2c15a-2f4f-4f6b-b29f-18d0a3898961\") " Oct 30 07:21:43.137493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.137462 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca" pod "9df2c15a-2f4f-4f6b-b29f-18d0a3898961" (UID: "9df2c15a-2f4f-4f6b-b29f-18d0a3898961"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:43.137995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.137969 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:43.139448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.139417 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca" pod "9df2c15a-2f4f-4f6b-b29f-18d0a3898961" (UID: "9df2c15a-2f4f-4f6b-b29f-18d0a3898961"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:21:43.139838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.139820 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h" pod "9df2c15a-2f4f-4f6b-b29f-18d0a3898961" (UID: "9df2c15a-2f4f-4f6b-b29f-18d0a3898961"). InnerVolumeSpecName "kube-api-access-qvn2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:43.239824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.239788 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:43.239824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.239816 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvn2h\" (UniqueName: \"kubernetes.io/projected/9df2c15a-2f4f-4f6b-b29f-18d0a3898961-kube-api-access-qvn2h\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:43.914975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.914944 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" Oct 30 07:21:43.915150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.914976 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc" event={"ID":"9df2c15a-2f4f-4f6b-b29f-18d0a3898961","Type":"ContainerDied","Data":"55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d"} Oct 30 07:21:43.915150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:43.915025 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55412ee6c8c2af61ee7ea7f54203dae9cb02418abd23c5b910f301f764efb11d" Oct 30 07:21:44.943942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:44.943895 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:46.811831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:46.811781 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:48.930467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:48.930417 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:49.046001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:49.045968 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:49.162501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:49.162453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:50.654780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:50.654749 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:52.963952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:52.963915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:53.854401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.854363 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg"] Oct 30 07:21:53.855008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.854994 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9df2c15a-2f4f-4f6b-b29f-18d0a3898961" containerName="revoke-keys" Oct 30 07:21:53.855064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.855009 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df2c15a-2f4f-4f6b-b29f-18d0a3898961" containerName="revoke-keys" Oct 30 07:21:53.855126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.855116 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df2c15a-2f4f-4f6b-b29f-18d0a3898961" containerName="revoke-keys" Oct 30 07:21:53.858832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.858810 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:53.890675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.886869 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:21:53.890675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.887019 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:21:53.891725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.891694 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg"] Oct 30 07:21:53.922249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.922218 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:21:53.922249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.922217 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:21:53.979443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.979411 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:53.979860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.979479 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxkgb\" (UniqueName: \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:53.979860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:53.979579 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.081418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.081385 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.081616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.081450 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxkgb\" (UniqueName: \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.081616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.081513 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.082160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.082139 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.083754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.083738 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.104950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.104880 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxkgb\" (UniqueName: \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\") pod \"maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.173839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.173805 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:54.480977 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:21:54.480947 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cb50f16_f233_4920_9079_bff1ef9c04b5.slice/crio-dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc WatchSource:0}: Error finding container dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc: Status 404 returned error can't find the container with id dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc Oct 30 07:21:54.481496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:54.481471 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg"] Oct 30 07:21:55.134679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:55.134565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:55.141575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:55.141548 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" event={"ID":"4cb50f16-f233-4920-9079-bff1ef9c04b5","Type":"ContainerStarted","Data":"0bf0e7ded0b68bee1cf572b119979993059b92523d602bb516c43e5c3042e87c"} Oct 30 07:21:55.141737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:55.141595 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" event={"ID":"4cb50f16-f233-4920-9079-bff1ef9c04b5","Type":"ContainerStarted","Data":"dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc"} Oct 30 07:21:55.184201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:55.184133 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" podStartSLOduration=2.184112706 podStartE2EDuration="2.184112706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:21:55.18268387 +0000 UTC m=+1478.873601308" watchObservedRunningTime="2026-10-30 07:21:55.184112706 +0000 UTC m=+1478.875030143" Oct 30 07:21:56.150296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:56.150265 5894 generic.go:356] "Generic (PLEG): container finished" podID="4cb50f16-f233-4920-9079-bff1ef9c04b5" containerID="0bf0e7ded0b68bee1cf572b119979993059b92523d602bb516c43e5c3042e87c" exitCode=0 Oct 30 07:21:56.150823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:56.150346 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" event={"ID":"4cb50f16-f233-4920-9079-bff1ef9c04b5","Type":"ContainerDied","Data":"0bf0e7ded0b68bee1cf572b119979993059b92523d602bb516c43e5c3042e87c"} Oct 30 07:21:56.930200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:56.930160 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:21:57.314236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.314209 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:57.339837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.339803 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\" (UniqueName: \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\") pod \"4cb50f16-f233-4920-9079-bff1ef9c04b5\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " Oct 30 07:21:57.340019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.339844 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\") pod \"4cb50f16-f233-4920-9079-bff1ef9c04b5\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " Oct 30 07:21:57.340019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.339991 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\") pod \"4cb50f16-f233-4920-9079-bff1ef9c04b5\" (UID: \"4cb50f16-f233-4920-9079-bff1ef9c04b5\") " Oct 30 07:21:57.340353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.340325 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca" pod "4cb50f16-f233-4920-9079-bff1ef9c04b5" (UID: "4cb50f16-f233-4920-9079-bff1ef9c04b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:21:57.340493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.340478 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:57.341950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.341926 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca" pod "4cb50f16-f233-4920-9079-bff1ef9c04b5" (UID: "4cb50f16-f233-4920-9079-bff1ef9c04b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:21:57.342024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.341982 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb" pod "4cb50f16-f233-4920-9079-bff1ef9c04b5" (UID: "4cb50f16-f233-4920-9079-bff1ef9c04b5"). InnerVolumeSpecName "kube-api-access-lxkgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:21:57.442077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.442038 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxkgb\" (UniqueName: \"kubernetes.io/projected/4cb50f16-f233-4920-9079-bff1ef9c04b5-kube-api-access-lxkgb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:57.442077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:57.442069 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cb50f16-f233-4920-9079-bff1ef9c04b5-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:21:58.168209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:58.168178 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" Oct 30 07:21:58.168427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:58.168206 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg" event={"ID":"4cb50f16-f233-4920-9079-bff1ef9c04b5","Type":"ContainerDied","Data":"dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc"} Oct 30 07:21:58.168427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:58.168254 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc940e08ad431372e5fa576d4160f8357f905c825f2fca56d4ace264403d0ebc" Oct 30 07:21:58.826163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:21:58.826125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:00.191485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:00.191452 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:00.373832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:00.373715 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:01.359237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:01.359205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:02.944253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:02.944217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:03.055329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:03.055295 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:03.179979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:03.179941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:03.568980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:03.568946 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:04.828629 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:22:04.828596 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:22:05.234763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:05.234681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:06.941316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:06.941277 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:07.342688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:07.342647 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:22:08.811495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:08.811461 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.265563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.265523 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.450554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.450521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.570477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.570442 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.820357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.820326 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.935297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.935204 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:10.967895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.967864 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc"] Oct 30 07:22:10.979605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:10.979574 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0cd8f0-75spc"] Oct 30 07:22:11.188942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:11.188859 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:11.492675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:11.492543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:12.431628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:12.431595 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df2c15a-2f4f-4f6b-b29f-18d0a3898961" path="/var/lib/kubelet/pods/9df2c15a-2f4f-4f6b-b29f-18d0a3898961/volumes" Oct 30 07:22:12.865739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:12.865693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:15.015939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:15.015908 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:16.841412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:16.841373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:17.509332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.509295 5894 scope.go:122] "RemoveContainer" containerID="3a4128d5e7d84182363e5bc71a80fbe0df2e1d686d24e0c92ed09de009f157b5" Oct 30 07:22:17.521022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.521000 5894 scope.go:122] "RemoveContainer" containerID="40ebf7ba32180c68d01626d0d19d9fc5a03f9cfdfc376a9e7cc941e3cf5f1802" Oct 30 07:22:17.532176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.532152 5894 scope.go:122] "RemoveContainer" containerID="4deea02f20f062e5ce313470ad8070cae474e5ae8c9c0d95693c553c96d36878" Oct 30 07:22:17.661619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.661582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:17.782182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.782152 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:17.952854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:17.952821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:18.087946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.087862 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:18.486078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.485883 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:18.609013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.608976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:18.612938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.612906 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b"] Oct 30 07:22:18.614075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.614049 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4cb50f16-f233-4920-9079-bff1ef9c04b5" containerName="revoke-keys" Oct 30 07:22:18.614075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.614072 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb50f16-f233-4920-9079-bff1ef9c04b5" containerName="revoke-keys" Oct 30 07:22:18.614283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.614269 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb50f16-f233-4920-9079-bff1ef9c04b5" containerName="revoke-keys" Oct 30 07:22:18.620491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.620465 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.668625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.668509 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:22:18.669049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.669025 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:22:18.669227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.669214 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:22:18.669414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.669399 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:22:18.680659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.676816 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b"] Oct 30 07:22:18.723719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.718179 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.724265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.724219 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6f2\" (UniqueName: \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.724502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.724470 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.837660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.827050 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.837660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.827198 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6f2\" (UniqueName: \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.837660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.827236 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.837660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.828276 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.837660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.837647 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:18.874734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:18.864760 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6f2\" (UniqueName: \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\") pod \"maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:19.016654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.015716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:19.020563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.017905 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:19.138349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.138247 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:19.269541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.269497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:19.393692 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:22:19.389700 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65da1960_a163_408c_8585_60a1d764fb36.slice/crio-eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd WatchSource:0}: Error finding container eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd: Status 404 returned error can't find the container with id eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd Oct 30 07:22:19.393692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.391477 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b"] Oct 30 07:22:19.411383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.408846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:19.983215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:19.983113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:20.113320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.113284 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg"] Oct 30 07:22:20.115954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.115925 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-438855-d6sqg"] Oct 30 07:22:20.126156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.126124 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:20.253973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.253879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:20.350629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.350589 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" event={"ID":"65da1960-a163-408c-8585-60a1d764fb36","Type":"ContainerStarted","Data":"25ce696ea742a5ddd1ed15c128efb3c9bf2329a1a18ba37764068ad219ac964f"} Oct 30 07:22:20.350873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.350675 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" event={"ID":"65da1960-a163-408c-8585-60a1d764fb36","Type":"ContainerStarted","Data":"eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd"} Oct 30 07:22:20.396398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.396347 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" podStartSLOduration=2.396331815 podStartE2EDuration="2.396331815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:20.393979954 +0000 UTC m=+1504.084897391" watchObservedRunningTime="2026-10-30 07:22:20.396331815 +0000 UTC m=+1504.087249251" Oct 30 07:22:20.433000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.432966 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb50f16-f233-4920-9079-bff1ef9c04b5" path="/var/lib/kubelet/pods/4cb50f16-f233-4920-9079-bff1ef9c04b5/volumes" Oct 30 07:22:20.629674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:20.629611 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:21.358921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:21.358889 5894 generic.go:356] "Generic (PLEG): container finished" podID="65da1960-a163-408c-8585-60a1d764fb36" containerID="25ce696ea742a5ddd1ed15c128efb3c9bf2329a1a18ba37764068ad219ac964f" exitCode=0 Oct 30 07:22:21.359323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:21.358983 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" event={"ID":"65da1960-a163-408c-8585-60a1d764fb36","Type":"ContainerDied","Data":"25ce696ea742a5ddd1ed15c128efb3c9bf2329a1a18ba37764068ad219ac964f"} Oct 30 07:22:22.160327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.160293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:22.551267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.551229 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:22.558801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.558778 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:22.648626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.648595 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\" (UniqueName: \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\") pod \"65da1960-a163-408c-8585-60a1d764fb36\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " Oct 30 07:22:22.648863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.648670 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\") pod \"65da1960-a163-408c-8585-60a1d764fb36\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " Oct 30 07:22:22.648863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.648731 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\") pod \"65da1960-a163-408c-8585-60a1d764fb36\" (UID: \"65da1960-a163-408c-8585-60a1d764fb36\") " Oct 30 07:22:22.649120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.649080 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca" pod "65da1960-a163-408c-8585-60a1d764fb36" (UID: "65da1960-a163-408c-8585-60a1d764fb36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:22.649534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.649488 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65da1960-a163-408c-8585-60a1d764fb36-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:22.651036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.651014 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2" pod "65da1960-a163-408c-8585-60a1d764fb36" (UID: "65da1960-a163-408c-8585-60a1d764fb36"). InnerVolumeSpecName "kube-api-access-gz6f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:22.651036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.651023 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca" pod "65da1960-a163-408c-8585-60a1d764fb36" (UID: "65da1960-a163-408c-8585-60a1d764fb36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:22.727074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.726980 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:22.751312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.751281 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65da1960-a163-408c-8585-60a1d764fb36-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:22.751312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:22.751310 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6f2\" (UniqueName: \"kubernetes.io/projected/65da1960-a163-408c-8585-60a1d764fb36-kube-api-access-gz6f2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:23.374102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:23.374076 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" Oct 30 07:22:23.374369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:23.374107 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b" event={"ID":"65da1960-a163-408c-8585-60a1d764fb36","Type":"ContainerDied","Data":"eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd"} Oct 30 07:22:23.374477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:23.374389 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeac65667cbc6319c9a64f6c53669c528c9773061b67d94fb565110c136051cd" Oct 30 07:22:23.520088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:23.520047 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:23.691806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:23.687586 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:24.461213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:24.461177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:24.645913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:24.645885 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:24.999989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:24.999933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:26.989667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:26.989599 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:27.105928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.105898 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:27.270974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.270878 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:27.435659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.434213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:27.904453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.904418 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5"] Oct 30 07:22:27.905371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.905342 5894 cpu_manager.go:414] "containerMap: removing container" podUID="65da1960-a163-408c-8585-60a1d764fb36" containerName="revoke-keys" Oct 30 07:22:27.905371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.905370 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65da1960-a163-408c-8585-60a1d764fb36" containerName="revoke-keys" Oct 30 07:22:27.905586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.905572 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="65da1960-a163-408c-8585-60a1d764fb36" containerName="revoke-keys" Oct 30 07:22:27.908956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.908934 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:27.928818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.928790 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5"] Oct 30 07:22:27.939249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.938152 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:22:27.939249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.938241 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:22:27.939249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.939196 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:22:27.939496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.939288 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:22:27.998616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.998573 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:27.999029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.998674 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:27.999029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:27.998780 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfqx9\" (UniqueName: \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.099686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.099658 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfqx9\" (UniqueName: \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.099870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.099855 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.099920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.099889 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.100502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.100477 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.102304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.102277 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.123363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.123331 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfqx9\" (UniqueName: \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\") pod \"maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.224970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.224873 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:28.531921 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:22:28.531886 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f047f8f_c948_47be_9638_f946a95adda8.slice/crio-df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4 WatchSource:0}: Error finding container df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4: Status 404 returned error can't find the container with id df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4 Oct 30 07:22:28.532059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:28.531936 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5"] Oct 30 07:22:29.422230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:29.422191 5894 generic.go:356] "Generic (PLEG): container finished" podID="1f047f8f-c948-47be-9638-f946a95adda8" containerID="684c79e892b19c91e4b2d527f00ad0d3173e9ef6e0a993874df7f4945bf1e333" exitCode=0 Oct 30 07:22:29.422610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:29.422232 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" event={"ID":"1f047f8f-c948-47be-9638-f946a95adda8","Type":"ContainerDied","Data":"684c79e892b19c91e4b2d527f00ad0d3173e9ef6e0a993874df7f4945bf1e333"} Oct 30 07:22:29.422610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:29.422300 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" event={"ID":"1f047f8f-c948-47be-9638-f946a95adda8","Type":"ContainerStarted","Data":"df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4"} Oct 30 07:22:29.549957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:29.549928 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:30.593855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.593828 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:30.637702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.637669 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\" (UniqueName: \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\") pod \"1f047f8f-c948-47be-9638-f946a95adda8\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " Oct 30 07:22:30.637889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.637808 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\") pod \"1f047f8f-c948-47be-9638-f946a95adda8\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " Oct 30 07:22:30.637889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.637852 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\") pod \"1f047f8f-c948-47be-9638-f946a95adda8\" (UID: \"1f047f8f-c948-47be-9638-f946a95adda8\") " Oct 30 07:22:30.638226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.638197 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca" pod "1f047f8f-c948-47be-9638-f946a95adda8" (UID: "1f047f8f-c948-47be-9638-f946a95adda8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:22:30.638501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.638479 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:30.640118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.640093 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9" pod "1f047f8f-c948-47be-9638-f946a95adda8" (UID: "1f047f8f-c948-47be-9638-f946a95adda8"). InnerVolumeSpecName "kube-api-access-vfqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:22:30.640413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.640392 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca" pod "1f047f8f-c948-47be-9638-f946a95adda8" (UID: "1f047f8f-c948-47be-9638-f946a95adda8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:22:30.739481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.739394 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfqx9\" (UniqueName: \"kubernetes.io/projected/1f047f8f-c948-47be-9638-f946a95adda8-kube-api-access-vfqx9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:30.739481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:30.739426 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f047f8f-c948-47be-9638-f946a95adda8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:22:31.217614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:31.217583 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:31.438469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:31.438430 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" event={"ID":"1f047f8f-c948-47be-9638-f946a95adda8","Type":"ContainerDied","Data":"df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4"} Oct 30 07:22:31.438469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:31.438463 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5" Oct 30 07:22:31.438758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:31.438488 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df9192937e81b45bc5d065fa857883fdc8bf3a1326ced716d5583e1b41d345d4" Oct 30 07:22:32.047434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:32.047394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:32.873566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:32.873489 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:34.387579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:34.387545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:34.829230 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:22:34.829186 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:22:36.005202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:36.005164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:37.472331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:37.472291 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:37.654169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:37.654110 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:37.812479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:37.812448 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:38.006226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.006195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:38.118351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.118269 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:38.150155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.150123 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b"] Oct 30 07:22:38.152590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.152567 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3d2270-qss5b"] Oct 30 07:22:38.231096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.231063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:38.432306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.432225 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65da1960-a163-408c-8585-60a1d764fb36" path="/var/lib/kubelet/pods/65da1960-a163-408c-8585-60a1d764fb36/volumes" Oct 30 07:22:38.761454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:38.761369 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:39.660535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:39.660499 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:39.807014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:39.806984 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:40.229126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:40.229096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:40.705992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:40.705947 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:40.867660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:40.864987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:42.930441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:42.930392 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:44.646777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:44.646738 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:44.660020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:44.659986 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:22:44.772517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:44.772481 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:45.073995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:45.073959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:45.218594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:45.218555 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:46.650302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:46.650272 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:46.776526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:46.776487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:47.185589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:47.185554 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:47.328795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:47.328748 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:47.528439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:47.528401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:49.341822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:49.341781 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:50.866303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:50.866266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:51.230869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:51.230779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:51.529987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:51.529954 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:51.741157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:51.741121 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:51.874864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:51.874771 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:52.376102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.376061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:52.602127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.602095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:52.727277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.727196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:52.771769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.771730 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5"] Oct 30 07:22:52.782662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.782608 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-efce7b-hjpp5"] Oct 30 07:22:52.856132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:52.856101 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:53.277806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:53.277763 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:54.324706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:54.324670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:54.436619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:54.436577 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f047f8f-c948-47be-9638-f946a95adda8" path="/var/lib/kubelet/pods/1f047f8f-c948-47be-9638-f946a95adda8/volumes" Oct 30 07:22:56.018740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:56.018705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:57.011469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.011387 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx"] Oct 30 07:22:57.012099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.012080 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1f047f8f-c948-47be-9638-f946a95adda8" containerName="revoke-keys" Oct 30 07:22:57.012161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.012101 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f047f8f-c948-47be-9638-f946a95adda8" containerName="revoke-keys" Oct 30 07:22:57.012229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.012219 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f047f8f-c948-47be-9638-f946a95adda8" containerName="revoke-keys" Oct 30 07:22:57.016967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.016945 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.035525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.035486 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx"] Oct 30 07:22:57.045039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.045001 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:22:57.045039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.045002 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:22:57.045270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.045025 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:22:57.045270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.045078 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:22:57.129949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.129911 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.130150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.130064 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.130236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.130214 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4rl\" (UniqueName: \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.231536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.231503 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.231737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.231601 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4rl\" (UniqueName: \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.231789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.231738 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.232375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.232351 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.234000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.233976 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.255036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.255009 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4rl\" (UniqueName: \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\") pod \"maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.331689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.331618 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:22:57.638759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.637789 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx"] Oct 30 07:22:57.638759 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:22:57.638013 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a671acd_9c5a_4197_9688_0f736ddffecd.slice/crio-b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88 WatchSource:0}: Error finding container b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88: Status 404 returned error can't find the container with id b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88 Oct 30 07:22:57.859248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:57.859212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:22:58.665304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:58.665265 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" event={"ID":"5a671acd-9c5a-4197-9688-0f736ddffecd","Type":"ContainerStarted","Data":"2289b5c80c8c7f0f0b0ca05eca903745aba0d39ff610a8097e0d7d31f51d3a8d"} Oct 30 07:22:58.665843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:58.665363 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" event={"ID":"5a671acd-9c5a-4197-9688-0f736ddffecd","Type":"ContainerStarted","Data":"b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88"} Oct 30 07:22:58.713059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:58.712985 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" podStartSLOduration=2.712964362 podStartE2EDuration="2.712964362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:22:58.71116353 +0000 UTC m=+1542.402080992" watchObservedRunningTime="2026-10-30 07:22:58.712964362 +0000 UTC m=+1542.403881801" Oct 30 07:22:59.674298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:59.674263 5894 generic.go:356] "Generic (PLEG): container finished" podID="5a671acd-9c5a-4197-9688-0f736ddffecd" containerID="2289b5c80c8c7f0f0b0ca05eca903745aba0d39ff610a8097e0d7d31f51d3a8d" exitCode=0 Oct 30 07:22:59.674727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:59.674355 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" event={"ID":"5a671acd-9c5a-4197-9688-0f736ddffecd","Type":"ContainerDied","Data":"2289b5c80c8c7f0f0b0ca05eca903745aba0d39ff610a8097e0d7d31f51d3a8d"} Oct 30 07:22:59.795542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:22:59.795505 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:00.328135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:00.328096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:00.463963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:00.463923 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:00.705779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:00.705745 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:00.989702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:00.986875 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:23:01.115777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.115739 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\") pod \"5a671acd-9c5a-4197-9688-0f736ddffecd\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " Oct 30 07:23:01.115981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.115836 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\" (UniqueName: \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\") pod \"5a671acd-9c5a-4197-9688-0f736ddffecd\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " Oct 30 07:23:01.115981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.115959 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\") pod \"5a671acd-9c5a-4197-9688-0f736ddffecd\" (UID: \"5a671acd-9c5a-4197-9688-0f736ddffecd\") " Oct 30 07:23:01.116348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.116324 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca" pod "5a671acd-9c5a-4197-9688-0f736ddffecd" (UID: "5a671acd-9c5a-4197-9688-0f736ddffecd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:01.116628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.116614 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:01.117940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.117916 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca" pod "5a671acd-9c5a-4197-9688-0f736ddffecd" (UID: "5a671acd-9c5a-4197-9688-0f736ddffecd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:01.118357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.118335 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl" pod "5a671acd-9c5a-4197-9688-0f736ddffecd" (UID: "5a671acd-9c5a-4197-9688-0f736ddffecd"). InnerVolumeSpecName "kube-api-access-ft4rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:01.218248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.218214 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a671acd-9c5a-4197-9688-0f736ddffecd-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:01.218248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.218244 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft4rl\" (UniqueName: \"kubernetes.io/projected/5a671acd-9c5a-4197-9688-0f736ddffecd-kube-api-access-ft4rl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:01.650197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.650108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:01.731000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.730969 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" Oct 30 07:23:01.731699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.730998 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx" event={"ID":"5a671acd-9c5a-4197-9688-0f736ddffecd","Type":"ContainerDied","Data":"b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88"} Oct 30 07:23:01.731923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:01.731897 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b676851141f321e30c14c009aa636283b1ac0a6a8d084c0b47edbb58fc552b88" Oct 30 07:23:02.196769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:02.196730 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:02.652457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:02.652423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:02.814194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:02.814164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:04.166422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:04.166389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:04.832453 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:23:04.832411 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:23:06.275298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:06.275259 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:06.428990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:06.428958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:06.560708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:06.559575 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:08.096482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:08.096448 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:08.264198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:08.264165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:08.462713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:08.451373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:08.600780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:08.600738 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:09.052724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:09.052691 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:23:09.052934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:09.052896 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" containerID="cri-o://4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d" gracePeriod=30 Oct 30 07:23:10.394906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:10.394872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:12.178159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:12.178116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:13.824546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:13.824508 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:13.953289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:13.953257 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:14.336194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.336127 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.408755 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.408822 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.408905 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.408962 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.409054 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.409127 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.409154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.409198 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.409664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.409322 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\") pod \"112f2bce-f857-46b6-acc7-293529163026\" (UID: \"112f2bce-f857-46b6-acc7-293529163026\") " Oct 30 07:23:14.457183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.457143 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:14.490656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.482965 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:14.490656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.483650 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:14.490656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.483845 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:14.490656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.483906 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:14.510655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.501677 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511704 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511763 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511781 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/112f2bce-f857-46b6-acc7-293529163026-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511794 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511807 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.513061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.511820 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/112f2bce-f857-46b6-acc7-293529163026-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.521747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.521447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:14.531755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.530675 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:14.533936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.533479 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:14.535667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.535610 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf" pod "112f2bce-f857-46b6-acc7-293529163026" (UID: "112f2bce-f857-46b6-acc7-293529163026"). InnerVolumeSpecName "kube-api-access-586kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:14.613630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.613535 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.613630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.613564 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/112f2bce-f857-46b6-acc7-293529163026-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.613630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.613574 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-586kf\" (UniqueName: \"kubernetes.io/projected/112f2bce-f857-46b6-acc7-293529163026-kube-api-access-586kf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:14.892222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.892133 5894 generic.go:356] "Generic (PLEG): container finished" podID="112f2bce-f857-46b6-acc7-293529163026" containerID="4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d" exitCode=0 Oct 30 07:23:14.892222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.892189 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" Oct 30 07:23:14.892798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.892215 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" event={"ID":"112f2bce-f857-46b6-acc7-293529163026","Type":"ContainerDied","Data":"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d"} Oct 30 07:23:14.892798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.892273 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr" event={"ID":"112f2bce-f857-46b6-acc7-293529163026","Type":"ContainerDied","Data":"917dad6c82a5fdca6ea8092ff6fa961eca0265288286aabeba5f205713f84329"} Oct 30 07:23:14.892798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.892299 5894 scope.go:122] "RemoveContainer" containerID="4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d" Oct 30 07:23:14.906001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.905982 5894 scope.go:122] "RemoveContainer" containerID="4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d" Oct 30 07:23:14.906354 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:23:14.906331 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d\": container with ID starting with 4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d not found: ID does not exist" containerID="4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d" Oct 30 07:23:14.906436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.906363 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d"} err="failed to get container status \"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d\": rpc error: code = NotFound desc = could not find container \"4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d\": container with ID starting with 4a93220c21571d444bf59ac4c867a712fc2054b3bb9808e06270b1e547dc631d not found: ID does not exist" Oct 30 07:23:14.958230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.958193 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:23:14.970557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:14.970526 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-b9zjr"] Oct 30 07:23:15.959687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:15.959651 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:23:15.960085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:15.959867 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" containerID="cri-o://f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e" gracePeriod=30 Oct 30 07:23:16.270486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.270407 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:23:16.270684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.270601 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" containerID="cri-o://9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d" gracePeriod=30 Oct 30 07:23:16.434414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.434384 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="112f2bce-f857-46b6-acc7-293529163026" path="/var/lib/kubelet/pods/112f2bce-f857-46b6-acc7-293529163026/volumes" Oct 30 07:23:16.698054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.698023 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:16.815237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.815201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:16.927957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:16.927919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:17.569998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:17.569967 5894 scope.go:122] "RemoveContainer" containerID="a0a57f8f4ca6e6ff13ab1bb349eefec15c8a93dc5009f0b0b386877dc3c75e96" Oct 30 07:23:17.583118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:17.583094 5894 scope.go:122] "RemoveContainer" containerID="01bf48a1142a617c18f5fe31c2454f0918ca7c8d47d15fd3196e4017a90539f1" Oct 30 07:23:17.595231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:17.595207 5894 scope.go:122] "RemoveContainer" containerID="33ebcf2ef9b86be9eb79c9365ff5dc7bb3bda56de260ad35473ead1286695ab8" Oct 30 07:23:17.606683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:17.606655 5894 scope.go:122] "RemoveContainer" containerID="0048ff9d84671d0e1c38efe2935f4413e2fd38e1180ed4ee6fc9b1d64a5c771d" Oct 30 07:23:18.437228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:18.437195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:20.144086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:20.144049 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:21.225165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.225097 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:23:21.308241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308213 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308337 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308413 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308454 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308524 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.308683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308664 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308718 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308752 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308788 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308841 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\") pod \"036ce68f-66d2-4260-aeb9-a60084fb2e10\" (UID: \"036ce68f-66d2-4260-aeb9-a60084fb2e10\") " Oct 30 07:23:21.308972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.308927 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.309249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309107 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.309249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309141 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:21.309249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309204 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.309497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309426 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.309497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309449 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.309497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309464 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.309497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309480 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036ce68f-66d2-4260-aeb9-a60084fb2e10-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.309756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.309534 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.310826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.310798 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:21.311075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.311053 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.311390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.311365 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:21.311527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.311487 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2" pod "036ce68f-66d2-4260-aeb9-a60084fb2e10" (UID: "036ce68f-66d2-4260-aeb9-a60084fb2e10"). InnerVolumeSpecName "kube-api-access-v5pw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:21.411371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.411340 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.411371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.411369 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.411562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.411391 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5pw2\" (UniqueName: \"kubernetes.io/projected/036ce68f-66d2-4260-aeb9-a60084fb2e10-kube-api-access-v5pw2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.411562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.411402 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036ce68f-66d2-4260-aeb9-a60084fb2e10-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.541448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.541420 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:23:21.613681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.613627 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.613845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.613765 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.613845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.613798 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.613845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.613829 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.613974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.613897 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614033 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.614064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614050 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614042 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.614182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614115 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614155 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614184 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614269 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\") pod \"4b7708c6-f652-4435-8ed2-4e36488ffd68\" (UID: \"4b7708c6-f652-4435-8ed2-4e36488ffd68\") " Oct 30 07:23:21.614467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614338 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:21.614555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.614476 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.615204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.615173 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.615328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.615209 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.615328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.615224 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.615328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.615239 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.615328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.615251 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b7708c6-f652-4435-8ed2-4e36488ffd68-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.616717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.616688 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "kube-api-access-w2rqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:21.616877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.616756 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:23:21.617267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.617248 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:21.617895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.617872 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo" pod "4b7708c6-f652-4435-8ed2-4e36488ffd68" (UID: "4b7708c6-f652-4435-8ed2-4e36488ffd68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:23:21.716495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.716462 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.716495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.716493 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.716747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.716508 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rqc\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-kube-api-access-w2rqc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.716747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.716518 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b7708c6-f652-4435-8ed2-4e36488ffd68-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:21.811116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.811029 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:21.942721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.942625 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerID="9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d" exitCode=0 Oct 30 07:23:21.942721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.942665 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" event={"ID":"4b7708c6-f652-4435-8ed2-4e36488ffd68","Type":"ContainerDied","Data":"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d"} Oct 30 07:23:21.942721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.942690 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" Oct 30 07:23:21.943001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.942733 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq" event={"ID":"4b7708c6-f652-4435-8ed2-4e36488ffd68","Type":"ContainerDied","Data":"240158716c5ec1c68830d704ae7314f514e60e7fb08495e71fbdaa7b02f0ac45"} Oct 30 07:23:21.943001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.942762 5894 scope.go:122] "RemoveContainer" containerID="9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d" Oct 30 07:23:21.944461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.944436 5894 generic.go:356] "Generic (PLEG): container finished" podID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerID="f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e" exitCode=0 Oct 30 07:23:21.944588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.944473 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" event={"ID":"036ce68f-66d2-4260-aeb9-a60084fb2e10","Type":"ContainerDied","Data":"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e"} Oct 30 07:23:21.944588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.944493 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" Oct 30 07:23:21.944588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.944515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj" event={"ID":"036ce68f-66d2-4260-aeb9-a60084fb2e10","Type":"ContainerDied","Data":"d5ae5d0e37c1f05426a7816dbe2f8eaf6987ce2ed43c31ebb8d4e2b8020f86b3"} Oct 30 07:23:21.953880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.953863 5894 scope.go:122] "RemoveContainer" containerID="9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d" Oct 30 07:23:21.954163 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:23:21.954143 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d\": container with ID starting with 9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d not found: ID does not exist" containerID="9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d" Oct 30 07:23:21.954234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.954173 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d"} err="failed to get container status \"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d\": rpc error: code = NotFound desc = could not find container \"9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d\": container with ID starting with 9ca77290dfb35bd3f9f337da924123e2186b97afdfb6adaceffb2b28ed1ffb2d not found: ID does not exist" Oct 30 07:23:21.954234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.954190 5894 scope.go:122] "RemoveContainer" containerID="f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e" Oct 30 07:23:21.963942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.963920 5894 scope.go:122] "RemoveContainer" containerID="f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e" Oct 30 07:23:21.964207 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:23:21.964192 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e\": container with ID starting with f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e not found: ID does not exist" containerID="f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e" Oct 30 07:23:21.964262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.964214 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e"} err="failed to get container status \"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e\": rpc error: code = NotFound desc = could not find container \"f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e\": container with ID starting with f4b811d506a22fa986610e99e2b50ceb3b59d77bb6168130efa4f824d5a1268e not found: ID does not exist" Oct 30 07:23:21.995718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.995690 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:23:21.999484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:21.999455 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-89tfj"] Oct 30 07:23:22.036728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:22.036684 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:23:22.039751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:22.039727 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-zt7tq"] Oct 30 07:23:22.431922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:22.431888 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" path="/var/lib/kubelet/pods/036ce68f-66d2-4260-aeb9-a60084fb2e10/volumes" Oct 30 07:23:22.432398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:22.432384 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" path="/var/lib/kubelet/pods/4b7708c6-f652-4435-8ed2-4e36488ffd68/volumes" Oct 30 07:23:22.650598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:22.650553 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:23:23.499731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:23.499701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:25.133601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:25.133571 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:26.818545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:26.818500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:27.398498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:27.398467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:28.729484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:28.729442 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:28.938531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:28.938468 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:29.055321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.055287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:29.261147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.261112 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:29.380963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.380871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:29.417250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.417222 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx"] Oct 30 07:23:29.428979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.428953 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ab3d83-gfpzx"] Oct 30 07:23:29.497247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.497208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:29.968505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:29.968474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:30.432131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:30.432101 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a671acd-9c5a-4197-9688-0f736ddffecd" path="/var/lib/kubelet/pods/5a671acd-9c5a-4197-9688-0f736ddffecd/volumes" Oct 30 07:23:31.354693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:31.354659 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:33.075101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:33.075057 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:34.788153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:34.788108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:34.850355 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:23:34.850314 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:23:36.075951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:36.075920 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:36.200241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:36.200205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:36.318234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:36.318201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:36.518676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:36.518530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:37.134294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:37.134260 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:37.261615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:37.261565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:37.372792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:37.372755 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:37.539069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:37.539035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:38.086566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.086534 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq"] Oct 30 07:23:38.087110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087097 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" Oct 30 07:23:38.087157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087111 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" Oct 30 07:23:38.087157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087124 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5a671acd-9c5a-4197-9688-0f736ddffecd" containerName="revoke-keys" Oct 30 07:23:38.087157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087129 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a671acd-9c5a-4197-9688-0f736ddffecd" containerName="revoke-keys" Oct 30 07:23:38.087157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087149 5894 cpu_manager.go:414] "containerMap: removing container" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" Oct 30 07:23:38.087157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087153 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087161 5894 cpu_manager.go:414] "containerMap: removing container" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087165 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087257 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="036ce68f-66d2-4260-aeb9-a60084fb2e10" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087265 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="112f2bce-f857-46b6-acc7-293529163026" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087279 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b7708c6-f652-4435-8ed2-4e36488ffd68" containerName="istio-proxy" Oct 30 07:23:38.087314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.087285 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a671acd-9c5a-4197-9688-0f736ddffecd" containerName="revoke-keys" Oct 30 07:23:38.090238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.090222 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.110360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.110323 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq"] Oct 30 07:23:38.120043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.120005 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:23:38.120043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.120005 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:23:38.120043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.120017 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:23:38.120348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.120087 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:23:38.167354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.167271 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.167807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.167478 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.167807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.167539 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbk7h\" (UniqueName: \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.269468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.269429 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.269705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.269499 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbk7h\" (UniqueName: \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.269705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.269696 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.270489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.270462 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.277064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.277032 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.292936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.292905 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbk7h\" (UniqueName: \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\") pod \"maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.403825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.403791 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:38.710294 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:23:38.710252 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2f2435_e256_43a2_a46d_66c931f71f30.slice/crio-aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637 WatchSource:0}: Error finding container aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637: Status 404 returned error can't find the container with id aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637 Oct 30 07:23:38.710943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:38.710924 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq"] Oct 30 07:23:39.075448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:39.075412 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" event={"ID":"dc2f2435-e256-43a2-a46d-66c931f71f30","Type":"ContainerStarted","Data":"cff64083dcfc52fc0053a56dc8775f6d640bd22aac0dfdaa7e9ffa2d90d32444"} Oct 30 07:23:39.075689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:39.075473 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" event={"ID":"dc2f2435-e256-43a2-a46d-66c931f71f30","Type":"ContainerStarted","Data":"aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637"} Oct 30 07:23:39.120161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:39.120065 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" podStartSLOduration=1.120038786 podStartE2EDuration="1.120038786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:23:39.118923548 +0000 UTC m=+1582.809840987" watchObservedRunningTime="2026-10-30 07:23:39.120038786 +0000 UTC m=+1582.810956224" Oct 30 07:23:39.644226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:39.644192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:40.084540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:40.084509 5894 generic.go:356] "Generic (PLEG): container finished" podID="dc2f2435-e256-43a2-a46d-66c931f71f30" containerID="cff64083dcfc52fc0053a56dc8775f6d640bd22aac0dfdaa7e9ffa2d90d32444" exitCode=0 Oct 30 07:23:40.084745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:40.084568 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" event={"ID":"dc2f2435-e256-43a2-a46d-66c931f71f30","Type":"ContainerDied","Data":"cff64083dcfc52fc0053a56dc8775f6d640bd22aac0dfdaa7e9ffa2d90d32444"} Oct 30 07:23:41.337886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.337860 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:41.424054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.423964 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\") pod \"dc2f2435-e256-43a2-a46d-66c931f71f30\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " Oct 30 07:23:41.424236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.424147 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\") pod \"dc2f2435-e256-43a2-a46d-66c931f71f30\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " Oct 30 07:23:41.424236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.424190 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\" (UniqueName: \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\") pod \"dc2f2435-e256-43a2-a46d-66c931f71f30\" (UID: \"dc2f2435-e256-43a2-a46d-66c931f71f30\") " Oct 30 07:23:41.424967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.424933 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca" pod "dc2f2435-e256-43a2-a46d-66c931f71f30" (UID: "dc2f2435-e256-43a2-a46d-66c931f71f30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:23:41.426222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.426192 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca" pod "dc2f2435-e256-43a2-a46d-66c931f71f30" (UID: "dc2f2435-e256-43a2-a46d-66c931f71f30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:23:41.426770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.426739 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h" pod "dc2f2435-e256-43a2-a46d-66c931f71f30" (UID: "dc2f2435-e256-43a2-a46d-66c931f71f30"). InnerVolumeSpecName "kube-api-access-pbk7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:23:41.527285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.527246 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:41.527285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.527284 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2f2435-e256-43a2-a46d-66c931f71f30-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:41.527482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:41.527299 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbk7h\" (UniqueName: \"kubernetes.io/projected/dc2f2435-e256-43a2-a46d-66c931f71f30-kube-api-access-pbk7h\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:23:42.042952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:42.042918 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:42.105696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:42.105669 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" Oct 30 07:23:42.105868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:42.105663 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq" event={"ID":"dc2f2435-e256-43a2-a46d-66c931f71f30","Type":"ContainerDied","Data":"aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637"} Oct 30 07:23:42.105868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:42.105792 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaec9d56df7ee8596152cdd98e3aae34ad47d144cfd59baec8002e445f683637" Oct 30 07:23:43.751652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:43.751610 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:45.622647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:45.622586 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:47.030081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:47.030044 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt"] Oct 30 07:23:47.042604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:47.042574 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2a2dbc4-6zdvt"] Oct 30 07:23:47.072002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:47.071967 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:47.240552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:47.240512 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:48.389195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:48.389157 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:48.431887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:48.431854 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3" path="/var/lib/kubelet/pods/413a5ac6-1ad2-41d0-8c1d-7ced23fc65e3/volumes" Oct 30 07:23:48.851055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:48.851021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:48.975876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:48.975842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:49.125702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:49.125562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:50.589603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:50.589567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:51.927203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:51.927164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:52.157698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.156714 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:52.271372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.271289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:52.672791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.672755 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:52.795990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.795959 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq"] Oct 30 07:23:52.799128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.799095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:52.808120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.808093 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-bdd496-r7jlq"] Oct 30 07:23:52.915658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:52.915221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:53.039438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:53.039399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:53.156254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:53.156220 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:53.342151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:53.342061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:54.432987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:54.432951 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2f2435-e256-43a2-a46d-66c931f71f30" path="/var/lib/kubelet/pods/dc2f2435-e256-43a2-a46d-66c931f71f30/volumes" Oct 30 07:23:54.759044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:54.758962 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:54.887999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:54.887947 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:55.005709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:55.005627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:56.579134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:56.579102 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:57.301250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:57.301205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:57.833985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:57.833941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:57.990722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:57.990690 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:58.639050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:58.639017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:58.990375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:58.990283 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:59.131237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:59.131204 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:23:59.370709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:23:59.370624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:00.993430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:00.991486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:01.129270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:01.129236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:01.290660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:01.290596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:01.432867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:01.432669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:02.199826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.199788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:02.310345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.310312 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:02.594070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.594032 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:24:02.594703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.594676 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dc2f2435-e256-43a2-a46d-66c931f71f30" containerName="revoke-keys" Oct 30 07:24:02.594703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.594701 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2f2435-e256-43a2-a46d-66c931f71f30" containerName="revoke-keys" Oct 30 07:24:02.594927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.594893 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2f2435-e256-43a2-a46d-66c931f71f30" containerName="revoke-keys" Oct 30 07:24:02.599854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.599829 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.632891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.632862 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-17bd11\"/\"ipp-route-17bd11-kserve-self-signed-certs\"" Oct 30 07:24:02.633855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.633830 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-17bd11\"/\"openshift-service-ca.crt\"" Oct 30 07:24:02.633855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.633849 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-17bd11\"/\"kube-root-ca.crt\"" Oct 30 07:24:02.634042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.633964 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-17bd11\"/\"default-dockercfg-tb9wm\"" Oct 30 07:24:02.635506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.635487 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:24:02.659858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.659821 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.660072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.659926 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.660072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.659982 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.660196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.660080 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.660234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.660209 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvj77\" (UniqueName: \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.660298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.660281 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.667761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.667729 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:02.761753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.761711 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvj77\" (UniqueName: \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.761959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.761861 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.761959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.761881 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.761959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.761926 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.762135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.761972 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.762135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.762042 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.762451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.762419 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.762597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.762465 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.762597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.762569 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.764616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.764590 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.764753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.764647 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.787341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.787288 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvj77\" (UniqueName: \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\") pod \"ipp-route-17bd11-kserve-795c875557-vcsfq\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:02.893740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.892120 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:02.919207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:02.918909 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:03.058077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.058045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:03.197115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.197023 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:03.284405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.284372 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:24:03.285987 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:24:03.285957 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97b32b94_135f_4181_94e2_be0ec3213b5c.slice/crio-1d22a4b55b97ceaac350ad517f7cfa89cd66166637c9b06c37fd34ec6929a796 WatchSource:0}: Error finding container 1d22a4b55b97ceaac350ad517f7cfa89cd66166637c9b06c37fd34ec6929a796: Status 404 returned error can't find the container with id 1d22a4b55b97ceaac350ad517f7cfa89cd66166637c9b06c37fd34ec6929a796 Oct 30 07:24:03.331919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.331877 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" event={"ID":"97b32b94-135f-4181-94e2-be0ec3213b5c","Type":"ContainerStarted","Data":"1d22a4b55b97ceaac350ad517f7cfa89cd66166637c9b06c37fd34ec6929a796"} Oct 30 07:24:03.401648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.401588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:03.779672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:03.778608 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:04.178381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:04.178287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:04.342149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:04.342114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" event={"ID":"97b32b94-135f-4181-94e2-be0ec3213b5c","Type":"ContainerStarted","Data":"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a"} Oct 30 07:24:04.386554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:04.386488 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" podStartSLOduration=2.386468448 podStartE2EDuration="2.386468448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:04.383988279 +0000 UTC m=+1608.074905717" watchObservedRunningTime="2026-10-30 07:24:04.386468448 +0000 UTC m=+1608.077385865" Oct 30 07:24:04.833477 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:24:04.833437 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:24:04.920422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:04.920383 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:04.943497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:04.943464 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:05.066866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:05.066827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:06.541030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:06.540983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:07.012835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:07.012752 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:07.141087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:07.141054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:08.537016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:08.536983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:08.776456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:08.776419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:08.905992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:08.905904 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:09.856045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:09.856013 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:24:10.338950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:10.338919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:10.753767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:10.753608 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:10.897182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:10.897150 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:12.113049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:12.113011 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:12.920355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:12.920319 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:12.933094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:12.933063 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:13.308335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.308301 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z"] Oct 30 07:24:13.312768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.312746 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.329331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.329289 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:24:13.333333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.333306 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z"] Oct 30 07:24:13.333613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.333592 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:24:13.341562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.341528 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:24:13.420664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.420622 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:24:13.429710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.429685 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:24:13.455056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.455013 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99cfs\" (UniqueName: \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.455056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.455061 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.455517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.455495 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.557045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.557003 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99cfs\" (UniqueName: \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.557247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.557050 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.557247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.557204 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.557770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.557739 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.559538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.559490 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.583127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.583095 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99cfs\" (UniqueName: \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.629028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.628995 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:13.764297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.764258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:13.931883 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:24:13.931847 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151a65e1_c917_4872_811a_8ce6c8dc6628.slice/crio-de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3 WatchSource:0}: Error finding container de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3: Status 404 returned error can't find the container with id de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3 Oct 30 07:24:13.932042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:13.932021 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z"] Oct 30 07:24:14.417066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:14.417030 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" event={"ID":"151a65e1-c917-4872-811a-8ce6c8dc6628","Type":"ContainerStarted","Data":"e8fbc22b7923235c6192443d5b20f5ffc73bb9c6ef743773c658d592b3ed4fef"} Oct 30 07:24:14.417567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:14.417092 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" event={"ID":"151a65e1-c917-4872-811a-8ce6c8dc6628","Type":"ContainerStarted","Data":"de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3"} Oct 30 07:24:14.460653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:14.460541 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" podStartSLOduration=1.460525359 podStartE2EDuration="1.460525359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:14.458050825 +0000 UTC m=+1618.148968262" watchObservedRunningTime="2026-10-30 07:24:14.460525359 +0000 UTC m=+1618.151442803" Oct 30 07:24:15.426339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:15.426306 5894 generic.go:356] "Generic (PLEG): container finished" podID="151a65e1-c917-4872-811a-8ce6c8dc6628" containerID="e8fbc22b7923235c6192443d5b20f5ffc73bb9c6ef743773c658d592b3ed4fef" exitCode=0 Oct 30 07:24:15.426757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:15.426385 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" event={"ID":"151a65e1-c917-4872-811a-8ce6c8dc6628","Type":"ContainerDied","Data":"e8fbc22b7923235c6192443d5b20f5ffc73bb9c6ef743773c658d592b3ed4fef"} Oct 30 07:24:15.566275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:15.566236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:16.605810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.605783 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:16.697921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.697838 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\") pod \"151a65e1-c917-4872-811a-8ce6c8dc6628\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " Oct 30 07:24:16.698093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.697982 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\" (UniqueName: \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\") pod \"151a65e1-c917-4872-811a-8ce6c8dc6628\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " Oct 30 07:24:16.698093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.698047 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\") pod \"151a65e1-c917-4872-811a-8ce6c8dc6628\" (UID: \"151a65e1-c917-4872-811a-8ce6c8dc6628\") " Oct 30 07:24:16.698607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.698578 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca" pod "151a65e1-c917-4872-811a-8ce6c8dc6628" (UID: "151a65e1-c917-4872-811a-8ce6c8dc6628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:16.700555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.700527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs" pod "151a65e1-c917-4872-811a-8ce6c8dc6628" (UID: "151a65e1-c917-4872-811a-8ce6c8dc6628"). InnerVolumeSpecName "kube-api-access-99cfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:16.700555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.700530 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca" pod "151a65e1-c917-4872-811a-8ce6c8dc6628" (UID: "151a65e1-c917-4872-811a-8ce6c8dc6628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:24:16.800433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.800396 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99cfs\" (UniqueName: \"kubernetes.io/projected/151a65e1-c917-4872-811a-8ce6c8dc6628-kube-api-access-99cfs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:16.800433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.800428 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:16.800433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:16.800438 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/151a65e1-c917-4872-811a-8ce6c8dc6628-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:17.295347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.295297 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:17.410643 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.410584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:17.446820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.446776 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" event={"ID":"151a65e1-c917-4872-811a-8ce6c8dc6628","Type":"ContainerDied","Data":"de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3"} Oct 30 07:24:17.447068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.446849 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6ff0be0ed2ac07b9a13696aa64921a9d199c6bd7479416a0adb714cf066cd3" Oct 30 07:24:17.447068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.446794 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z" Oct 30 07:24:17.663240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.663152 5894 scope.go:122] "RemoveContainer" containerID="60f367a3ce6911c49533a837aff494a48889d8d3684e80d35ca61393a37755c4" Oct 30 07:24:17.673789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.673742 5894 scope.go:122] "RemoveContainer" containerID="ddf9897fb244a56fc553c46731a4b0930f8454bfbf0b0301861eb97498d5ecbc" Oct 30 07:24:17.683570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:17.683547 5894 scope.go:122] "RemoveContainer" containerID="60b93ee319fb3db348620489238f835b4967d987698d29e5c27d9c5d819cc1d0" Oct 30 07:24:19.105340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:19.105313 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:20.698907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:20.698866 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:21.374549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:21.374517 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:24:21.374762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:21.374735 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" containerID="cri-o://0941b7b5657a87f88680d20d202d87406e5c0b4bc48bcb290768b22c2e1bcd31" gracePeriod=30 Oct 30 07:24:22.341829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:22.341793 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:24.565408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:24.565368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:26.515539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.515480 5894 generic.go:356] "Generic (PLEG): container finished" podID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerID="0941b7b5657a87f88680d20d202d87406e5c0b4bc48bcb290768b22c2e1bcd31" exitCode=0 Oct 30 07:24:26.516205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.515787 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" event={"ID":"61c9448e-f07e-4cfc-af5a-a1edee7b4a55","Type":"ContainerDied","Data":"0941b7b5657a87f88680d20d202d87406e5c0b4bc48bcb290768b22c2e1bcd31"} Oct 30 07:24:26.622837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.622790 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:26.753178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.753098 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:24:26.779900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.779865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:26.837604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837548 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.837604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837605 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.837883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837653 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.837883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837703 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.837883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837799 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.837883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837852 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.838109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837925 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:26.838109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.837956 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.838109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.838035 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.838109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.838093 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\") pod \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\" (UID: \"61c9448e-f07e-4cfc-af5a-a1edee7b4a55\") " Oct 30 07:24:26.838677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.838611 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.839125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.839081 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:26.839125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.839100 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:26.839328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.839161 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:26.839328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.839272 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:26.844555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.844529 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:24:26.845024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.844998 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:24:26.845876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.845842 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:26.847491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.847462 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk" pod "61c9448e-f07e-4cfc-af5a-a1edee7b4a55" (UID: "61c9448e-f07e-4cfc-af5a-a1edee7b4a55"). InnerVolumeSpecName "kube-api-access-hnmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:26.901656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.900228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:26.939856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939816 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnmtk\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-kube-api-access-hnmtk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.939856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939855 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939872 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939885 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939898 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939912 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939924 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:26.940152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:26.939936 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c9448e-f07e-4cfc-af5a-a1edee7b4a55-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:27.136432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.136397 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:27.492236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.492143 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z"] Oct 30 07:24:27.498107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.498077 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc546d-x984z"] Oct 30 07:24:27.523929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.523901 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" Oct 30 07:24:27.524372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.523899 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd" event={"ID":"61c9448e-f07e-4cfc-af5a-a1edee7b4a55","Type":"ContainerDied","Data":"eb43439b9974428d1bf0094bb00a043ca073936a66748f2b2d62bea6c70b6de5"} Oct 30 07:24:27.524372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.524022 5894 scope.go:122] "RemoveContainer" containerID="0941b7b5657a87f88680d20d202d87406e5c0b4bc48bcb290768b22c2e1bcd31" Oct 30 07:24:27.536306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.536274 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:27.608019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.607987 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:24:27.620246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.620220 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-2wvtd"] Oct 30 07:24:27.707952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.707798 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:27.836264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:27.836232 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:28.043469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:28.043434 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:28.431648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:28.431591 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="151a65e1-c917-4872-811a-8ce6c8dc6628" path="/var/lib/kubelet/pods/151a65e1-c917-4872-811a-8ce6c8dc6628/volumes" Oct 30 07:24:28.432174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:28.432155 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" path="/var/lib/kubelet/pods/61c9448e-f07e-4cfc-af5a-a1edee7b4a55/volumes" Oct 30 07:24:29.418257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:29.418219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:31.702595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:31.701005 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:31.822045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:31.822010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:31.945964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:31.945929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:33.442964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:33.442926 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:34.828076 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:24:34.828036 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:24:34.955347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:34.955316 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:35.086130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:35.086010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:35.359745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:35.359580 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:35.486062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:35.486019 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:37.019447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:37.019414 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:38.668552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:38.668519 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:40.247469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:40.247435 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:41.182673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.182623 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh"] Oct 30 07:24:41.183256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183242 5894 cpu_manager.go:414] "containerMap: removing container" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" Oct 30 07:24:41.183302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183258 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" Oct 30 07:24:41.183302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183278 5894 cpu_manager.go:414] "containerMap: removing container" podUID="151a65e1-c917-4872-811a-8ce6c8dc6628" containerName="revoke-keys" Oct 30 07:24:41.183302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183283 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="151a65e1-c917-4872-811a-8ce6c8dc6628" containerName="revoke-keys" Oct 30 07:24:41.183400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183363 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c9448e-f07e-4cfc-af5a-a1edee7b4a55" containerName="istio-proxy" Oct 30 07:24:41.183400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.183374 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="151a65e1-c917-4872-811a-8ce6c8dc6628" containerName="revoke-keys" Oct 30 07:24:41.186348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.186330 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.203436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.203380 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:24:41.215921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.215898 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:24:41.216073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.215986 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:24:41.216073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.216035 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:24:41.219864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.219836 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh"] Oct 30 07:24:41.285608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.285573 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgccw\" (UniqueName: \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.285608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.285627 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.286099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.285742 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.387355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.387305 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgccw\" (UniqueName: \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.387542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.387367 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.387542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.387417 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.388149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.388121 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.389908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.389880 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.411184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.411154 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgccw\" (UniqueName: \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\") pod \"maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.499952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.499875 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:41.801965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.801920 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh"] Oct 30 07:24:41.802363 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:24:41.802340 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747ba449_8d93_4ff6_b334_611fc495bc51.slice/crio-67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f WatchSource:0}: Error finding container 67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f: Status 404 returned error can't find the container with id 67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f Oct 30 07:24:41.942537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:41.942480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:42.651692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:42.651396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" event={"ID":"747ba449-8d93-4ff6-b334-611fc495bc51","Type":"ContainerStarted","Data":"763c91d27a179ab56993b0d6240715532d6350fde7b1a5eed57fc6b036c23891"} Oct 30 07:24:42.651692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:42.651452 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" event={"ID":"747ba449-8d93-4ff6-b334-611fc495bc51","Type":"ContainerStarted","Data":"67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f"} Oct 30 07:24:42.693316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:42.693267 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" podStartSLOduration=1.693251036 podStartE2EDuration="1.693251036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:24:42.691726657 +0000 UTC m=+1646.382644094" watchObservedRunningTime="2026-10-30 07:24:42.693251036 +0000 UTC m=+1646.384168472" Oct 30 07:24:43.635105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:43.635071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:43.659034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:43.659001 5894 generic.go:356] "Generic (PLEG): container finished" podID="747ba449-8d93-4ff6-b334-611fc495bc51" containerID="763c91d27a179ab56993b0d6240715532d6350fde7b1a5eed57fc6b036c23891" exitCode=0 Oct 30 07:24:43.659478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:43.659092 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" event={"ID":"747ba449-8d93-4ff6-b334-611fc495bc51","Type":"ContainerDied","Data":"763c91d27a179ab56993b0d6240715532d6350fde7b1a5eed57fc6b036c23891"} Oct 30 07:24:44.811563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.811539 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:44.934563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.934485 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\" (UniqueName: \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\") pod \"747ba449-8d93-4ff6-b334-611fc495bc51\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " Oct 30 07:24:44.934563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.934553 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\") pod \"747ba449-8d93-4ff6-b334-611fc495bc51\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " Oct 30 07:24:44.934772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.934625 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\") pod \"747ba449-8d93-4ff6-b334-611fc495bc51\" (UID: \"747ba449-8d93-4ff6-b334-611fc495bc51\") " Oct 30 07:24:44.934946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.934924 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca" pod "747ba449-8d93-4ff6-b334-611fc495bc51" (UID: "747ba449-8d93-4ff6-b334-611fc495bc51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:44.935038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.935024 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:44.936506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.936484 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca" pod "747ba449-8d93-4ff6-b334-611fc495bc51" (UID: "747ba449-8d93-4ff6-b334-611fc495bc51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:24:44.936668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:44.936648 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw" pod "747ba449-8d93-4ff6-b334-611fc495bc51" (UID: "747ba449-8d93-4ff6-b334-611fc495bc51"). InnerVolumeSpecName "kube-api-access-qgccw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:45.036464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.036430 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/747ba449-8d93-4ff6-b334-611fc495bc51-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:45.036464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.036460 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgccw\" (UniqueName: \"kubernetes.io/projected/747ba449-8d93-4ff6-b334-611fc495bc51-kube-api-access-qgccw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:45.393870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.393823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:45.672953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.672877 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" Oct 30 07:24:45.672953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.672874 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh" event={"ID":"747ba449-8d93-4ff6-b334-611fc495bc51","Type":"ContainerDied","Data":"67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f"} Oct 30 07:24:45.673132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:45.672997 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67673fb4da379f6d11425f9d647c5e9e72deb722dcfa3544311d486a3768968f" Oct 30 07:24:47.153777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:47.153734 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:24:47.207030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:47.206991 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:48.937932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:48.937899 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:49.136789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:49.136757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:51.226900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:51.226856 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:53.014404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.014367 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t"] Oct 30 07:24:53.014993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.014976 5894 cpu_manager.go:414] "containerMap: removing container" podUID="747ba449-8d93-4ff6-b334-611fc495bc51" containerName="revoke-keys" Oct 30 07:24:53.014993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.014993 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747ba449-8d93-4ff6-b334-611fc495bc51" containerName="revoke-keys" Oct 30 07:24:53.015129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.015078 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="747ba449-8d93-4ff6-b334-611fc495bc51" containerName="revoke-keys" Oct 30 07:24:53.019230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.019209 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.039412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.039378 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t"] Oct 30 07:24:53.047998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.047961 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:24:53.047998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.047967 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:24:53.048198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.048045 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.048198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.048066 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:24:53.048299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.048233 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.048374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.048349 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf6g\" (UniqueName: \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.048479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.048456 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:24:53.149946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.149907 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.150151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.150029 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.150151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.150072 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf6g\" (UniqueName: \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.150760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.150732 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.152847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.152823 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.173779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.173752 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf6g\" (UniqueName: \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\") pod \"maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.334971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.334936 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:53.469416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.469381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:53.641259 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:24:53.641228 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66b73c0_f959_4068_a998_6856cc32dba1.slice/crio-5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39 WatchSource:0}: Error finding container 5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39: Status 404 returned error can't find the container with id 5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39 Oct 30 07:24:53.641454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.641418 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t"] Oct 30 07:24:53.743337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:53.743303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" event={"ID":"f66b73c0-f959-4068-a998-6856cc32dba1","Type":"ContainerStarted","Data":"5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39"} Oct 30 07:24:54.751041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:54.751010 5894 generic.go:356] "Generic (PLEG): container finished" podID="f66b73c0-f959-4068-a998-6856cc32dba1" containerID="f3c80d7c52c8c030dec1da2e2495b540aee62e8a879a0dd223aca49361a9dc00" exitCode=0 Oct 30 07:24:54.751462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:54.751103 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" event={"ID":"f66b73c0-f959-4068-a998-6856cc32dba1","Type":"ContainerDied","Data":"f3c80d7c52c8c030dec1da2e2495b540aee62e8a879a0dd223aca49361a9dc00"} Oct 30 07:24:55.352038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.352001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:55.917068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.917040 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:55.995446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.995406 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\") pod \"f66b73c0-f959-4068-a998-6856cc32dba1\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " Oct 30 07:24:55.995622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.995565 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\" (UniqueName: \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\") pod \"f66b73c0-f959-4068-a998-6856cc32dba1\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " Oct 30 07:24:55.995806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.995726 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\") pod \"f66b73c0-f959-4068-a998-6856cc32dba1\" (UID: \"f66b73c0-f959-4068-a998-6856cc32dba1\") " Oct 30 07:24:55.995872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.995796 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca" pod "f66b73c0-f959-4068-a998-6856cc32dba1" (UID: "f66b73c0-f959-4068-a998-6856cc32dba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:24:55.996448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.996422 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:55.997820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.997785 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g" pod "f66b73c0-f959-4068-a998-6856cc32dba1" (UID: "f66b73c0-f959-4068-a998-6856cc32dba1"). InnerVolumeSpecName "kube-api-access-4pf6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:24:55.998019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:55.997991 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca" pod "f66b73c0-f959-4068-a998-6856cc32dba1" (UID: "f66b73c0-f959-4068-a998-6856cc32dba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:24:56.098204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:56.098171 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pf6g\" (UniqueName: \"kubernetes.io/projected/f66b73c0-f959-4068-a998-6856cc32dba1-kube-api-access-4pf6g\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:56.098204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:56.098202 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66b73c0-f959-4068-a998-6856cc32dba1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:24:56.767706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:56.767615 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" Oct 30 07:24:56.767706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:56.767647 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t" event={"ID":"f66b73c0-f959-4068-a998-6856cc32dba1","Type":"ContainerDied","Data":"5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39"} Oct 30 07:24:56.767706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:56.767701 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b961ffab22957ad1678e4fc246593d46d9cf1b9e58130932d12cbc25b423a39" Oct 30 07:24:57.824391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:57.824357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:59.236731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:59.236699 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:24:59.536355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:24:59.536324 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:00.249893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:00.249858 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:25:00.250358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:00.250048 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" containerID="cri-o://7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b" gracePeriod=30 Oct 30 07:25:01.020719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.020678 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82"] Oct 30 07:25:01.021278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.021263 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f66b73c0-f959-4068-a998-6856cc32dba1" containerName="revoke-keys" Oct 30 07:25:01.021278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.021278 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66b73c0-f959-4068-a998-6856cc32dba1" containerName="revoke-keys" Oct 30 07:25:01.021421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.021360 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66b73c0-f959-4068-a998-6856cc32dba1" containerName="revoke-keys" Oct 30 07:25:01.025266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.025242 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.041254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.041229 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:25:01.053756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.053719 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:25:01.053977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.053731 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:25:01.053977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.053818 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:25:01.056586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.056561 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82"] Oct 30 07:25:01.175324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.175289 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg96g\" (UniqueName: \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.175505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.175408 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.175505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.175429 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.277564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.277528 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg96g\" (UniqueName: \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.278016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.277683 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.278016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.277707 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.278341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.278323 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.280201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.280172 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.305328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.305298 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg96g\" (UniqueName: \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\") pod \"maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.339172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.339135 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:01.350419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.350385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:01.645967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.645935 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82"] Oct 30 07:25:01.646711 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:25:01.646682 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f271e8_9371_44ce_9a0f_19ed4ab727cd.slice/crio-48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb WatchSource:0}: Error finding container 48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb: Status 404 returned error can't find the container with id 48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb Oct 30 07:25:01.802720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:01.802685 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" event={"ID":"53f271e8-9371-44ce-9a0f-19ed4ab727cd","Type":"ContainerStarted","Data":"48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb"} Oct 30 07:25:02.811871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:02.811836 5894 generic.go:356] "Generic (PLEG): container finished" podID="53f271e8-9371-44ce-9a0f-19ed4ab727cd" containerID="89145c483aa316fe6d81283cf5b31df57aaa853befaccac7152846f07a775fe4" exitCode=0 Oct 30 07:25:02.812356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:02.811919 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" event={"ID":"53f271e8-9371-44ce-9a0f-19ed4ab727cd","Type":"ContainerDied","Data":"89145c483aa316fe6d81283cf5b31df57aaa853befaccac7152846f07a775fe4"} Oct 30 07:25:03.113332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:03.113297 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:03.236242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:03.236205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:03.788061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:03.788028 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:03.903752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:03.903564 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:03.991739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:03.991711 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:04.119490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.119402 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\") pod \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " Oct 30 07:25:04.119490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.119454 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\" (UniqueName: \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\") pod \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " Oct 30 07:25:04.119490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.119489 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\") pod \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\" (UID: \"53f271e8-9371-44ce-9a0f-19ed4ab727cd\") " Oct 30 07:25:04.119973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.119942 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca" pod "53f271e8-9371-44ce-9a0f-19ed4ab727cd" (UID: "53f271e8-9371-44ce-9a0f-19ed4ab727cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:04.121703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.121670 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca" pod "53f271e8-9371-44ce-9a0f-19ed4ab727cd" (UID: "53f271e8-9371-44ce-9a0f-19ed4ab727cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:04.121825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.121794 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g" pod "53f271e8-9371-44ce-9a0f-19ed4ab727cd" (UID: "53f271e8-9371-44ce-9a0f-19ed4ab727cd"). InnerVolumeSpecName "kube-api-access-vg96g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:04.221457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.221424 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:04.221457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.221459 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg96g\" (UniqueName: \"kubernetes.io/projected/53f271e8-9371-44ce-9a0f-19ed4ab727cd-kube-api-access-vg96g\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:04.221687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.221475 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f271e8-9371-44ce-9a0f-19ed4ab727cd-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:04.320348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.320223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:04.481234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.481173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:04.601089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.601045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:04.811674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.811181 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:04.830589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.829885 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" event={"ID":"53f271e8-9371-44ce-9a0f-19ed4ab727cd","Type":"ContainerDied","Data":"48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb"} Oct 30 07:25:04.830589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.829934 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d9cbc1b90068124e54a3cd4daa43b6892019fe1daedb1d46643192b86730eb" Oct 30 07:25:04.830589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:04.829988 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82" Oct 30 07:25:04.840845 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:25:04.840795 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:25:05.054079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.054046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:05.183310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.183213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:05.533976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.533951 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:25:05.662343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662302 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.662506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662430 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.662593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662572 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.662730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662707 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.662878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662754 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.662878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662806 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:05.662878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662818 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:05.662878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662818 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.663082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662888 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.663082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.662927 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.663082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663000 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\") pod \"3be0c87d-1f01-4978-af59-f863085abf27\" (UID: \"3be0c87d-1f01-4978-af59-f863085abf27\") " Oct 30 07:25:05.663082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663031 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:05.663277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663205 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:05.663655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663617 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.663737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663681 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.663737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663696 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.663737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663710 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.663831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.663768 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:05.664890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.664861 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:25:05.665107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.665079 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "kube-api-access-ztxbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:05.665415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.665394 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:05.665798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.665782 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy" pod "3be0c87d-1f01-4978-af59-f863085abf27" (UID: "3be0c87d-1f01-4978-af59-f863085abf27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:25:05.765105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.765019 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3be0c87d-1f01-4978-af59-f863085abf27-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.765105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.765052 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3be0c87d-1f01-4978-af59-f863085abf27-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.765105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.765061 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3be0c87d-1f01-4978-af59-f863085abf27-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.765105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.765069 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.765105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.765077 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxbf\" (UniqueName: \"kubernetes.io/projected/3be0c87d-1f01-4978-af59-f863085abf27-kube-api-access-ztxbf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:05.838889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.838852 5894 generic.go:356] "Generic (PLEG): container finished" podID="3be0c87d-1f01-4978-af59-f863085abf27" containerID="7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b" exitCode=0 Oct 30 07:25:05.839095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.838928 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" Oct 30 07:25:05.839095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.838939 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" event={"ID":"3be0c87d-1f01-4978-af59-f863085abf27","Type":"ContainerDied","Data":"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b"} Oct 30 07:25:05.839095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.839029 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76" event={"ID":"3be0c87d-1f01-4978-af59-f863085abf27","Type":"ContainerDied","Data":"99de004d6532634557372fd33a289374750ce95a7e2174106a636f38413e281b"} Oct 30 07:25:05.839095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.839056 5894 scope.go:122] "RemoveContainer" containerID="7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b" Oct 30 07:25:05.861160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.861137 5894 scope.go:122] "RemoveContainer" containerID="7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b" Oct 30 07:25:05.861463 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:25:05.861440 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b\": container with ID starting with 7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b not found: ID does not exist" containerID="7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b" Oct 30 07:25:05.861560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.861474 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b"} err="failed to get container status \"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b\": rpc error: code = NotFound desc = could not find container \"7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b\": container with ID starting with 7c8fb9667f5304f47a5f8f3c21bbf0ada4add93dd47b1c34c7e95f0678ca9d2b not found: ID does not exist" Oct 30 07:25:05.894674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.894604 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:25:05.906394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:05.906363 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-c2m76"] Oct 30 07:25:06.432520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:06.432487 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be0c87d-1f01-4978-af59-f863085abf27" path="/var/lib/kubelet/pods/3be0c87d-1f01-4978-af59-f863085abf27/volumes" Oct 30 07:25:06.902213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:06.902173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:08.649693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:08.649656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:09.086602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:09.086568 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:09.237842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:09.237724 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:10.608383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:10.608348 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:10.816201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:10.816165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:10.943825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:10.943730 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:11.591228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:11.591194 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:11.819155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:11.819120 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:12.325393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:12.325346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:12.933499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:12.933469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:14.327263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:14.327223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:14.498122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:14.498063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:14.618394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:14.618312 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:14.917747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:14.888039 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:15.074213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:15.074168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:15.090619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:15.090576 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh"] Oct 30 07:25:15.102684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:15.102653 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a8f428-vgkfh"] Oct 30 07:25:15.187852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:15.187716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:15.611175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:15.611136 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:16.432546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:16.432510 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747ba449-8d93-4ff6-b334-611fc495bc51" path="/var/lib/kubelet/pods/747ba449-8d93-4ff6-b334-611fc495bc51/volumes" Oct 30 07:25:17.553429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:17.553394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:17.727409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:17.727380 5894 scope.go:122] "RemoveContainer" containerID="632499a4bf5c4eb3c39750d68eab18aae9edae1c6967ea3774f08cb0983a35fe" Oct 30 07:25:17.737481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:17.737458 5894 scope.go:122] "RemoveContainer" containerID="5816bdf3c810da2bee479adfc6e0bf9213676e2af6cb46db86822519c8d90d4b" Oct 30 07:25:17.746359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:17.746335 5894 scope.go:122] "RemoveContainer" containerID="6dc4aa19b7b79c95528738e7c2fef07db4aa0cad62457fca9b1f5690890c34f5" Oct 30 07:25:19.320834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:19.320800 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:21.483570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:21.483536 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:21.617989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:21.617951 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:21.740841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:21.740757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:22.158656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:22.158600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:23.353191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:23.353160 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:23.626287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:23.626184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:23.899172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:23.899083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:24.015593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.015560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:24.146543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.146500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:24.148674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.148644 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t"] Oct 30 07:25:24.161172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.161101 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-96ca87-ph64t"] Oct 30 07:25:24.292517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.292479 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:24.435898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.435809 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66b73c0-f959-4068-a998-6856cc32dba1" path="/var/lib/kubelet/pods/f66b73c0-f959-4068-a998-6856cc32dba1/volumes" Oct 30 07:25:24.646124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:24.646089 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:25.918798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:25.918763 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:26.632305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:26.632270 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:25:27.849520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:27.849483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:27.988867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:27.988826 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:29.690262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:29.690208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:29.812264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:29.812228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:29.958952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:29.958875 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:31.376126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:31.376088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:31.500779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:31.500746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:31.694260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:31.694171 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:31.813963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:31.813931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:33.444920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:33.444887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:33.653291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:33.653255 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:33.924884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:33.924851 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:34.829159 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:25:34.829119 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:25:34.888955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.888919 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76"] Oct 30 07:25:34.889784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.889758 5894 cpu_manager.go:414] "containerMap: removing container" podUID="53f271e8-9371-44ce-9a0f-19ed4ab727cd" containerName="revoke-keys" Oct 30 07:25:34.889784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.889784 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f271e8-9371-44ce-9a0f-19ed4ab727cd" containerName="revoke-keys" Oct 30 07:25:34.889957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.889810 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" Oct 30 07:25:34.889957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.889819 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" Oct 30 07:25:34.890027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.890000 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f271e8-9371-44ce-9a0f-19ed4ab727cd" containerName="revoke-keys" Oct 30 07:25:34.890027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.890014 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be0c87d-1f01-4978-af59-f863085abf27" containerName="istio-proxy" Oct 30 07:25:34.896036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.896010 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:34.904199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.904174 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:34.904306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.904222 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:34.904306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.904254 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl2x2\" (UniqueName: \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:34.911322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.911296 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:25:34.914213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.914188 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76"] Oct 30 07:25:34.923898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.923870 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:25:34.924069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.923917 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:25:34.924069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:34.923947 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:25:35.005988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.005950 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl2x2\" (UniqueName: \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.006220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.006200 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.006263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.006252 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.006932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.006910 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.008489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.008464 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.030603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.030571 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl2x2\" (UniqueName: \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\") pod \"maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.210768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.210689 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:35.348833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.348794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:35.518981 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:25:35.518950 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbe8c93_ab30_439e_a290_6017db8b4434.slice/crio-24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8 WatchSource:0}: Error finding container 24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8: Status 404 returned error can't find the container with id 24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8 Oct 30 07:25:35.519995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:35.519970 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76"] Oct 30 07:25:36.107519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:36.107280 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" event={"ID":"3dbe8c93-ab30-439e-a290-6017db8b4434","Type":"ContainerStarted","Data":"1ad62cd2a7dac4597bcbb88f0a73d3be77d7a06828bd16ccc70b8b0d2b5bf96e"} Oct 30 07:25:36.107519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:36.107381 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" event={"ID":"3dbe8c93-ab30-439e-a290-6017db8b4434","Type":"ContainerStarted","Data":"24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8"} Oct 30 07:25:36.149772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:36.149723 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" podStartSLOduration=2.149709662 podStartE2EDuration="2.149709662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:36.147859402 +0000 UTC m=+1699.838776841" watchObservedRunningTime="2026-10-30 07:25:36.149709662 +0000 UTC m=+1699.840627098" Oct 30 07:25:37.121160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.121113 5894 generic.go:356] "Generic (PLEG): container finished" podID="3dbe8c93-ab30-439e-a290-6017db8b4434" containerID="1ad62cd2a7dac4597bcbb88f0a73d3be77d7a06828bd16ccc70b8b0d2b5bf96e" exitCode=0 Oct 30 07:25:37.121702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.121476 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" event={"ID":"3dbe8c93-ab30-439e-a290-6017db8b4434","Type":"ContainerDied","Data":"1ad62cd2a7dac4597bcbb88f0a73d3be77d7a06828bd16ccc70b8b0d2b5bf96e"} Oct 30 07:25:37.195625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.195588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:37.307936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.307901 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:37.317069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.317037 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82"] Oct 30 07:25:37.320356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.320321 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1beb92-sbk82"] Oct 30 07:25:37.444507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.444425 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q"] Oct 30 07:25:37.448467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.448439 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.469045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.469012 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q"] Oct 30 07:25:37.536846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.536811 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.537033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.536869 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.537033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.536905 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtznh\" (UniqueName: \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.638587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.638552 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.638832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.638623 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.638832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.638716 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtznh\" (UniqueName: \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.639363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.639340 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.640982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.640952 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.662090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.662060 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtznh\" (UniqueName: \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\") pod \"maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.762574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.762490 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:37.868844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:37.867846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:38.068020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.067992 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q"] Oct 30 07:25:38.069092 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:25:38.069050 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a0c30f_9268_4379_b683_674aef2482d7.slice/crio-5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2 WatchSource:0}: Error finding container 5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2: Status 404 returned error can't find the container with id 5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2 Oct 30 07:25:38.130202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.130165 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" event={"ID":"75a0c30f-9268-4379-b683-674aef2482d7","Type":"ContainerStarted","Data":"5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2"} Oct 30 07:25:38.276521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.276494 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:38.347050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.346960 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\") pod \"3dbe8c93-ab30-439e-a290-6017db8b4434\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " Oct 30 07:25:38.347050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.347010 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\" (UniqueName: \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\") pod \"3dbe8c93-ab30-439e-a290-6017db8b4434\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " Oct 30 07:25:38.347292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.347078 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\") pod \"3dbe8c93-ab30-439e-a290-6017db8b4434\" (UID: \"3dbe8c93-ab30-439e-a290-6017db8b4434\") " Oct 30 07:25:38.347519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.347494 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca" pod "3dbe8c93-ab30-439e-a290-6017db8b4434" (UID: "3dbe8c93-ab30-439e-a290-6017db8b4434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:38.349483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.349455 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca" pod "3dbe8c93-ab30-439e-a290-6017db8b4434" (UID: "3dbe8c93-ab30-439e-a290-6017db8b4434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:38.349483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.349480 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2" pod "3dbe8c93-ab30-439e-a290-6017db8b4434" (UID: "3dbe8c93-ab30-439e-a290-6017db8b4434"). InnerVolumeSpecName "kube-api-access-sl2x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:38.432512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.432482 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f271e8-9371-44ce-9a0f-19ed4ab727cd" path="/var/lib/kubelet/pods/53f271e8-9371-44ce-9a0f-19ed4ab727cd/volumes" Oct 30 07:25:38.449155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.449123 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:38.449155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.449151 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl2x2\" (UniqueName: \"kubernetes.io/projected/3dbe8c93-ab30-439e-a290-6017db8b4434-kube-api-access-sl2x2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:38.449346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:38.449164 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbe8c93-ab30-439e-a290-6017db8b4434-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:39.118263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.118232 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:39.137173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.137142 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" event={"ID":"3dbe8c93-ab30-439e-a290-6017db8b4434","Type":"ContainerDied","Data":"24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8"} Oct 30 07:25:39.137595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.137196 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d5a92747492515053343ab4fcbe242629e61872bf461fd6f31c34081bdccf8" Oct 30 07:25:39.137595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.137150 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76" Oct 30 07:25:39.139151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.139118 5894 generic.go:356] "Generic (PLEG): container finished" podID="75a0c30f-9268-4379-b683-674aef2482d7" containerID="01e0a844f8fd45eea1b0f726fdc8c66720efc05920b9a528ca9287056d5aaa63" exitCode=0 Oct 30 07:25:39.139284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:39.139254 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" event={"ID":"75a0c30f-9268-4379-b683-674aef2482d7","Type":"ContainerDied","Data":"01e0a844f8fd45eea1b0f726fdc8c66720efc05920b9a528ca9287056d5aaa63"} Oct 30 07:25:40.303348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.303317 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:40.374592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.374553 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\") pod \"75a0c30f-9268-4379-b683-674aef2482d7\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " Oct 30 07:25:40.374780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.374757 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\") pod \"75a0c30f-9268-4379-b683-674aef2482d7\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " Oct 30 07:25:40.374963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.374940 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\" (UniqueName: \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\") pod \"75a0c30f-9268-4379-b683-674aef2482d7\" (UID: \"75a0c30f-9268-4379-b683-674aef2482d7\") " Oct 30 07:25:40.375115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.375086 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca" pod "75a0c30f-9268-4379-b683-674aef2482d7" (UID: "75a0c30f-9268-4379-b683-674aef2482d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:40.375542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.375509 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:40.376919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.376891 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca" pod "75a0c30f-9268-4379-b683-674aef2482d7" (UID: "75a0c30f-9268-4379-b683-674aef2482d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:40.377247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.377224 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh" pod "75a0c30f-9268-4379-b683-674aef2482d7" (UID: "75a0c30f-9268-4379-b683-674aef2482d7"). InnerVolumeSpecName "kube-api-access-qtznh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:40.476988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.476952 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtznh\" (UniqueName: \"kubernetes.io/projected/75a0c30f-9268-4379-b683-674aef2482d7-kube-api-access-qtznh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:40.476988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:40.476982 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75a0c30f-9268-4379-b683-674aef2482d7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:41.154573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:41.154544 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" Oct 30 07:25:41.154573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:41.154550 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q" event={"ID":"75a0c30f-9268-4379-b683-674aef2482d7","Type":"ContainerDied","Data":"5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2"} Oct 30 07:25:41.154866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:41.154600 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5838838eaa2648be32d3a8917a44d77b945ef0d5b51402e0dd1e07758d3d73d2" Oct 30 07:25:41.417621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:41.417532 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:43.041242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:43.041198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:43.824712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:43.824669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:44.093520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.093426 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:44.231421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.231389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:44.847761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.847726 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c"] Oct 30 07:25:44.848341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848323 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3dbe8c93-ab30-439e-a290-6017db8b4434" containerName="revoke-keys" Oct 30 07:25:44.848444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848343 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbe8c93-ab30-439e-a290-6017db8b4434" containerName="revoke-keys" Oct 30 07:25:44.848444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848381 5894 cpu_manager.go:414] "containerMap: removing container" podUID="75a0c30f-9268-4379-b683-674aef2482d7" containerName="revoke-keys" Oct 30 07:25:44.848444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848389 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a0c30f-9268-4379-b683-674aef2482d7" containerName="revoke-keys" Oct 30 07:25:44.848603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848562 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbe8c93-ab30-439e-a290-6017db8b4434" containerName="revoke-keys" Oct 30 07:25:44.848603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.848581 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a0c30f-9268-4379-b683-674aef2482d7" containerName="revoke-keys" Oct 30 07:25:44.851882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.851861 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:44.868767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.868737 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:25:44.881148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.881123 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:25:44.881148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.881131 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:25:44.881509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.881493 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:25:44.884185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.884162 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c"] Oct 30 07:25:44.943903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.943868 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:44.944074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.944059 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:44.944146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:44.944094 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78m7n\" (UniqueName: \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.045601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.045567 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.045779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.045751 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.045823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.045784 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78m7n\" (UniqueName: \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.046360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.046338 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.048132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.048105 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.069404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.069366 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78m7n\" (UniqueName: \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\") pod \"maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.167552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.167476 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:45.476350 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:25:45.476302 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1acef0c_b9ee_471d_9b67_690b349ad9c0.slice/crio-8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540 WatchSource:0}: Error finding container 8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540: Status 404 returned error can't find the container with id 8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540 Oct 30 07:25:45.476474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.476461 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c"] Oct 30 07:25:45.538421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:45.538391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:46.200375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:46.200275 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" event={"ID":"c1acef0c-b9ee-471d-9b67-690b349ad9c0","Type":"ContainerStarted","Data":"ddc41a192343f6950162bff76c552577dee889a8eb536d3243120aeb489a8ca2"} Oct 30 07:25:46.200375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:46.200347 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" event={"ID":"c1acef0c-b9ee-471d-9b67-690b349ad9c0","Type":"ContainerStarted","Data":"8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540"} Oct 30 07:25:46.243199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:46.243149 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" podStartSLOduration=2.243134245 podStartE2EDuration="2.243134245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:25:46.240750706 +0000 UTC m=+1709.931668142" watchObservedRunningTime="2026-10-30 07:25:46.243134245 +0000 UTC m=+1709.934051709" Oct 30 07:25:47.208856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:47.208818 5894 generic.go:356] "Generic (PLEG): container finished" podID="c1acef0c-b9ee-471d-9b67-690b349ad9c0" containerID="ddc41a192343f6950162bff76c552577dee889a8eb536d3243120aeb489a8ca2" exitCode=0 Oct 30 07:25:47.209262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:47.208908 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" event={"ID":"c1acef0c-b9ee-471d-9b67-690b349ad9c0","Type":"ContainerDied","Data":"ddc41a192343f6950162bff76c552577dee889a8eb536d3243120aeb489a8ca2"} Oct 30 07:25:47.228364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:47.228330 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:48.076423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.076386 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:48.189452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.189408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:48.366228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.366204 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:48.504335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.504303 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\" (UniqueName: \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\") pod \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " Oct 30 07:25:48.504520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.504408 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\") pod \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " Oct 30 07:25:48.504520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.504428 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\") pod \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\" (UID: \"c1acef0c-b9ee-471d-9b67-690b349ad9c0\") " Oct 30 07:25:48.504796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.504763 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca" pod "c1acef0c-b9ee-471d-9b67-690b349ad9c0" (UID: "c1acef0c-b9ee-471d-9b67-690b349ad9c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:25:48.505236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.505214 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:48.506584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.506562 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n" pod "c1acef0c-b9ee-471d-9b67-690b349ad9c0" (UID: "c1acef0c-b9ee-471d-9b67-690b349ad9c0"). InnerVolumeSpecName "kube-api-access-78m7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:25:48.506760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.506743 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca" pod "c1acef0c-b9ee-471d-9b67-690b349ad9c0" (UID: "c1acef0c-b9ee-471d-9b67-690b349ad9c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:25:48.609720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.607140 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78m7n\" (UniqueName: \"kubernetes.io/projected/c1acef0c-b9ee-471d-9b67-690b349ad9c0-kube-api-access-78m7n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:48.609720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.607182 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1acef0c-b9ee-471d-9b67-690b349ad9c0-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:25:48.924675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:48.924561 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:49.133444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:49.133398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:49.223692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:49.223596 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" Oct 30 07:25:49.223692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:49.223627 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c" event={"ID":"c1acef0c-b9ee-471d-9b67-690b349ad9c0","Type":"ContainerDied","Data":"8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540"} Oct 30 07:25:49.223692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:49.223689 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e86ffadd941b52865d4a6266f3957d67094feb2cdaf49dd7bd761a5e08c0540" Oct 30 07:25:49.253897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:49.253871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:50.138396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:50.138366 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:25:50.661074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:50.661037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:50.774501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:50.774467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:52.367389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:52.367349 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:54.022681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:54.022613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:55.495328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:55.495301 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:55.614386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:55.614352 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:55.732181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:55.732147 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:57.221526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:57.221442 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:58.826624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:58.826573 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:59.347127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:59.347090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:59.475941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:59.475898 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:25:59.484239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:59.484206 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q"] Oct 30 07:25:59.496410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:59.496384 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-3a5866-b8c7q"] Oct 30 07:25:59.604406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:25:59.604314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:00.004783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:00.004692 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:00.120837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:00.120793 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:00.434519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:00.434481 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a0c30f-9268-4379-b683-674aef2482d7" path="/var/lib/kubelet/pods/75a0c30f-9268-4379-b683-674aef2482d7/volumes" Oct 30 07:26:01.304030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:01.303989 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:03.383986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.383948 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:03.721211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.721119 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc"] Oct 30 07:26:03.723494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.721852 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c1acef0c-b9ee-471d-9b67-690b349ad9c0" containerName="revoke-keys" Oct 30 07:26:03.723754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.723733 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1acef0c-b9ee-471d-9b67-690b349ad9c0" containerName="revoke-keys" Oct 30 07:26:03.724321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.724070 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1acef0c-b9ee-471d-9b67-690b349ad9c0" containerName="revoke-keys" Oct 30 07:26:03.728231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.728210 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.759925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.759893 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:03.760172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.760156 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:03.760358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.760344 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:03.760530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.760514 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:03.760976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.760948 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc"] Oct 30 07:26:03.864605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.864568 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.872036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.871959 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk9ng\" (UniqueName: \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.872355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.872335 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.890229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.890182 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:03.992733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.992630 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.992733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.992713 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk9ng\" (UniqueName: \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.992989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.992756 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.993418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.993398 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:03.995053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:03.995031 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:04.022160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.022124 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk9ng\" (UniqueName: \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\") pod \"maas-api-revoke-sub-0893d6e3-855sc\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:04.105811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.105776 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:04.125163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.125128 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c"] Oct 30 07:26:04.127766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.127738 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-210755-mgp9c"] Oct 30 07:26:04.192964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.192925 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:04.230401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.230368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:04.432381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.432350 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1acef0c-b9ee-471d-9b67-690b349ad9c0" path="/var/lib/kubelet/pods/c1acef0c-b9ee-471d-9b67-690b349ad9c0/volumes" Oct 30 07:26:04.517880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.515120 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc"] Oct 30 07:26:04.517880 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:04.515183 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e45d8d4_4662_4bbd_b218_a4fb338f2b23.slice/crio-49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826 WatchSource:0}: Error finding container 49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826: Status 404 returned error can't find the container with id 49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826 Oct 30 07:26:04.664549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:04.664515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:04.831235 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:26:04.831196 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:26:05.348574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:05.348536 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" event={"ID":"0e45d8d4-4662-4bbd-b218-a4fb338f2b23","Type":"ContainerStarted","Data":"2bc843af641086df5605d1f24aeea44745a9b212b70cd3ec57ed304055375a89"} Oct 30 07:26:05.348823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:05.348601 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" event={"ID":"0e45d8d4-4662-4bbd-b218-a4fb338f2b23","Type":"ContainerStarted","Data":"49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826"} Oct 30 07:26:05.394027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:05.393975 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" podStartSLOduration=2.393958474 podStartE2EDuration="2.393958474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:05.392356919 +0000 UTC m=+1729.083274357" watchObservedRunningTime="2026-10-30 07:26:05.393958474 +0000 UTC m=+1729.084875912" Oct 30 07:26:06.030239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.030199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:06.160375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.160326 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:06.364242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.364154 5894 generic.go:356] "Generic (PLEG): container finished" podID="0e45d8d4-4662-4bbd-b218-a4fb338f2b23" containerID="2bc843af641086df5605d1f24aeea44745a9b212b70cd3ec57ed304055375a89" exitCode=0 Oct 30 07:26:06.364425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.364239 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" event={"ID":"0e45d8d4-4662-4bbd-b218-a4fb338f2b23","Type":"ContainerDied","Data":"2bc843af641086df5605d1f24aeea44745a9b212b70cd3ec57ed304055375a89"} Oct 30 07:26:06.711484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.711401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:06.848679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:06.845365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:07.091579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.091543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:07.531486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.531461 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:07.557070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.557042 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\" (UniqueName: \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\") pod \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " Oct 30 07:26:07.557233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.557137 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\") pod \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " Oct 30 07:26:07.557233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.557205 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\") pod \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\" (UID: \"0e45d8d4-4662-4bbd-b218-a4fb338f2b23\") " Oct 30 07:26:07.557596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.557572 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca" pod "0e45d8d4-4662-4bbd-b218-a4fb338f2b23" (UID: "0e45d8d4-4662-4bbd-b218-a4fb338f2b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:07.557914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.557893 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:07.559257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.559233 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng" pod "0e45d8d4-4662-4bbd-b218-a4fb338f2b23" (UID: "0e45d8d4-4662-4bbd-b218-a4fb338f2b23"). InnerVolumeSpecName "kube-api-access-wk9ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:07.559257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.559249 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca" pod "0e45d8d4-4662-4bbd-b218-a4fb338f2b23" (UID: "0e45d8d4-4662-4bbd-b218-a4fb338f2b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:07.658995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.658902 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:07.658995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.658930 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk9ng\" (UniqueName: \"kubernetes.io/projected/0e45d8d4-4662-4bbd-b218-a4fb338f2b23-kube-api-access-wk9ng\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:07.734896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:07.734858 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:08.379059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:08.379033 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" Oct 30 07:26:08.379059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:08.379058 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc" event={"ID":"0e45d8d4-4662-4bbd-b218-a4fb338f2b23","Type":"ContainerDied","Data":"49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826"} Oct 30 07:26:08.379489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:08.379107 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49db53ca936e77c36321d23bc6c6b5e07793c5137d255b7413a8a2ed322b9826" Oct 30 07:26:09.539316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:09.539280 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:11.234380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:11.234346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:11.356314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:11.356279 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:11.645530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:11.645492 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:12.085221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.085188 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl"] Oct 30 07:26:12.085796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.085780 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0e45d8d4-4662-4bbd-b218-a4fb338f2b23" containerName="revoke-keys" Oct 30 07:26:12.085796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.085797 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e45d8d4-4662-4bbd-b218-a4fb338f2b23" containerName="revoke-keys" Oct 30 07:26:12.085899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.085891 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e45d8d4-4662-4bbd-b218-a4fb338f2b23" containerName="revoke-keys" Oct 30 07:26:12.089550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.089528 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.117907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.117873 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:12.118132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.118113 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:12.120780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.120750 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl"] Oct 30 07:26:12.121762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.121733 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-649tm\" (UniqueName: \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.121898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.121810 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.121898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.121848 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.143178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.143149 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:12.143356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.143238 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:12.223741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.223702 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-649tm\" (UniqueName: \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.223915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.223798 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.223915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.223835 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.224460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.224438 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.226486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.226459 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.247369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.247336 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-649tm\" (UniqueName: \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\") pod \"maas-api-revoke-sub-837f9426-ghhnl\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.405886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.405807 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:12.716683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:12.716653 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl"] Oct 30 07:26:12.717672 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:12.717612 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4dff8e_8550_409c_a7da_e50eb01a55ef.slice/crio-27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e WatchSource:0}: Error finding container 27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e: Status 404 returned error can't find the container with id 27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e Oct 30 07:26:13.128537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:13.128502 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:13.431260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:13.431226 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" event={"ID":"8d4dff8e-8550-409c-a7da-e50eb01a55ef","Type":"ContainerStarted","Data":"4fc31b8ba67f8aa8a973015c68bf95e51812e53c7c859c29921835271bfcfb4f"} Oct 30 07:26:13.431692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:13.431279 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" event={"ID":"8d4dff8e-8550-409c-a7da-e50eb01a55ef","Type":"ContainerStarted","Data":"27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e"} Oct 30 07:26:13.474710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:13.474653 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" podStartSLOduration=1.474626499 podStartE2EDuration="1.474626499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:13.471687741 +0000 UTC m=+1737.162605180" watchObservedRunningTime="2026-10-30 07:26:13.474626499 +0000 UTC m=+1737.165543935" Oct 30 07:26:13.740053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:13.739969 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:14.439618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:14.439583 5894 generic.go:356] "Generic (PLEG): container finished" podID="8d4dff8e-8550-409c-a7da-e50eb01a55ef" containerID="4fc31b8ba67f8aa8a973015c68bf95e51812e53c7c859c29921835271bfcfb4f" exitCode=0 Oct 30 07:26:14.440052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:14.439659 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" event={"ID":"8d4dff8e-8550-409c-a7da-e50eb01a55ef","Type":"ContainerDied","Data":"4fc31b8ba67f8aa8a973015c68bf95e51812e53c7c859c29921835271bfcfb4f"} Oct 30 07:26:14.844797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:14.844761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:15.034385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.034358 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:15.553683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.552996 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:15.734282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.734208 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:15.783845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.783729 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\") pod \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " Oct 30 07:26:15.783845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.783838 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\" (UniqueName: \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\") pod \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " Oct 30 07:26:15.784155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.783974 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\") pod \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\" (UID: \"8d4dff8e-8550-409c-a7da-e50eb01a55ef\") " Oct 30 07:26:15.785319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.785283 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca" pod "8d4dff8e-8550-409c-a7da-e50eb01a55ef" (UID: "8d4dff8e-8550-409c-a7da-e50eb01a55ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:15.787980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.787953 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca" pod "8d4dff8e-8550-409c-a7da-e50eb01a55ef" (UID: "8d4dff8e-8550-409c-a7da-e50eb01a55ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:15.788094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.787987 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm" pod "8d4dff8e-8550-409c-a7da-e50eb01a55ef" (UID: "8d4dff8e-8550-409c-a7da-e50eb01a55ef"). InnerVolumeSpecName "kube-api-access-649tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:15.898689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.889150 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:15.898689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.889194 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-649tm\" (UniqueName: \"kubernetes.io/projected/8d4dff8e-8550-409c-a7da-e50eb01a55ef-kube-api-access-649tm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:15.898689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.889210 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d4dff8e-8550-409c-a7da-e50eb01a55ef-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:15.906915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:15.906870 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:16.456966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:16.456937 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" Oct 30 07:26:16.457129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:16.456936 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl" event={"ID":"8d4dff8e-8550-409c-a7da-e50eb01a55ef","Type":"ContainerDied","Data":"27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e"} Oct 30 07:26:16.457129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:16.457070 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27440928eb10ce2428a8e4361be34df99daffec349f5e089b1858eb72fa0eb7e" Oct 30 07:26:16.723077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:16.722978 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:17.790133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:17.790102 5894 scope.go:122] "RemoveContainer" containerID="c4bb07b42fb6beeab53dc585a190443a58ceb6e9ba6f61743e62b39ab93eff9c" Oct 30 07:26:17.800540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:17.800517 5894 scope.go:122] "RemoveContainer" containerID="1bd7354ecc4a04e62db2be662e192dde481924bec2f0591dd90c14bcbcf1b83c" Oct 30 07:26:17.809473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:17.809443 5894 scope.go:122] "RemoveContainer" containerID="fa14cae109660e5caee3d2ebd6ec6b0177a4de16c09b3fb6962ba78c2bbe4223" Oct 30 07:26:18.655956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.655919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:18.666055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.666023 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk"] Oct 30 07:26:18.667046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.667019 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8d4dff8e-8550-409c-a7da-e50eb01a55ef" containerName="revoke-keys" Oct 30 07:26:18.667046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.667044 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4dff8e-8550-409c-a7da-e50eb01a55ef" containerName="revoke-keys" Oct 30 07:26:18.667281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.667199 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4dff8e-8550-409c-a7da-e50eb01a55ef" containerName="revoke-keys" Oct 30 07:26:18.674813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.674779 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.695232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.695200 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk"] Oct 30 07:26:18.699349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.699297 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:18.699524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.699419 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:18.699524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.699492 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:18.704415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.704371 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:18.728043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.728012 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.728232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.728064 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp5ht\" (UniqueName: \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.728232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.728216 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.830072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.830033 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp5ht\" (UniqueName: \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.830588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.830115 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.830588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.830257 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.830957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.830933 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.832745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.832711 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.853313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.853284 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp5ht\" (UniqueName: \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\") pod \"maas-api-revoke-sub-28b9cf88-78zjk\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:18.997165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:18.997091 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:19.265776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:19.265702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:19.314810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:19.314783 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk"] Oct 30 07:26:19.314998 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:19.314970 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb7152b_25cd_436c_a0a1_9ac2f0afe187.slice/crio-604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce WatchSource:0}: Error finding container 604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce: Status 404 returned error can't find the container with id 604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce Oct 30 07:26:19.416254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:19.416221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:19.480336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:19.480304 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" event={"ID":"fbb7152b-25cd-436c-a0a1-9ac2f0afe187","Type":"ContainerStarted","Data":"604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce"} Oct 30 07:26:20.455518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:20.455428 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:20.509915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:20.509885 5894 generic.go:356] "Generic (PLEG): container finished" podID="fbb7152b-25cd-436c-a0a1-9ac2f0afe187" containerID="2f89a2812faf317b8ddfb127fbd8bc84732d156bc1cc2a6b2dd2db80685f41f8" exitCode=0 Oct 30 07:26:20.510071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:20.509964 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" event={"ID":"fbb7152b-25cd-436c-a0a1-9ac2f0afe187","Type":"ContainerDied","Data":"2f89a2812faf317b8ddfb127fbd8bc84732d156bc1cc2a6b2dd2db80685f41f8"} Oct 30 07:26:20.981794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:20.981752 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:21.160233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.160192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:21.692978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.692948 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:21.772509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.772426 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\") pod \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " Oct 30 07:26:21.772713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.772517 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\") pod \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " Oct 30 07:26:21.772713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.772558 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\" (UniqueName: \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\") pod \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\" (UID: \"fbb7152b-25cd-436c-a0a1-9ac2f0afe187\") " Oct 30 07:26:21.772857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.772837 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca" pod "fbb7152b-25cd-436c-a0a1-9ac2f0afe187" (UID: "fbb7152b-25cd-436c-a0a1-9ac2f0afe187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:21.773056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.773037 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:21.774801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.774763 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht" pod "fbb7152b-25cd-436c-a0a1-9ac2f0afe187" (UID: "fbb7152b-25cd-436c-a0a1-9ac2f0afe187"). InnerVolumeSpecName "kube-api-access-rp5ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:21.775024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.775000 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca" pod "fbb7152b-25cd-436c-a0a1-9ac2f0afe187" (UID: "fbb7152b-25cd-436c-a0a1-9ac2f0afe187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:21.874720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.874679 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:21.874720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:21.874711 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp5ht\" (UniqueName: \"kubernetes.io/projected/fbb7152b-25cd-436c-a0a1-9ac2f0afe187-kube-api-access-rp5ht\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:22.528889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.528857 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" Oct 30 07:26:22.528889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.528876 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk" event={"ID":"fbb7152b-25cd-436c-a0a1-9ac2f0afe187","Type":"ContainerDied","Data":"604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce"} Oct 30 07:26:22.529070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.528936 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="604e9317b2542a1a26538e754b0562badb2b72f9bcc8fc925b7ae0ee43dbe8ce" Oct 30 07:26:22.594886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.594851 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:22.757692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.757567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:22.902981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:22.902936 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:23.215858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:23.215823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:23.353419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:23.353385 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:26:23.394378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:23.394336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:23.414334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:23.414295 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76"] Oct 30 07:26:23.416541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:23.416519 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-17bd11-lzc76"] Oct 30 07:26:24.103412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:24.103377 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:24.435141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:24.435042 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbe8c93-ab30-439e-a290-6017db8b4434" path="/var/lib/kubelet/pods/3dbe8c93-ab30-439e-a290-6017db8b4434/volumes" Oct 30 07:26:24.651884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:24.651830 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:26:24.652795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:24.652760 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" containerID="cri-o://90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387" gracePeriod=30 Oct 30 07:26:26.054623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.054564 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6"] Oct 30 07:26:26.059664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.057906 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fbb7152b-25cd-436c-a0a1-9ac2f0afe187" containerName="revoke-keys" Oct 30 07:26:26.059664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.058102 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb7152b-25cd-436c-a0a1-9ac2f0afe187" containerName="revoke-keys" Oct 30 07:26:26.067448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.067419 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb7152b-25cd-436c-a0a1-9ac2f0afe187" containerName="revoke-keys" Oct 30 07:26:26.072018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.071970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:26.165678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.163716 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.204669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.194006 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6"] Oct 30 07:26:26.252415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.216048 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:26.252708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.216191 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:26.266111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.259353 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.266111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.259500 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h7km\" (UniqueName: \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.266111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.259675 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.314922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.314845 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:26.315240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.315209 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:26.360448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.360411 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.360628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.360509 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h7km\" (UniqueName: \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.360628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.360627 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.361261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.361235 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.367322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.367285 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.387678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.385959 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h7km\" (UniqueName: \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\") pod \"maas-api-revoke-sub-bf1eb026-fhfs6\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.636209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.636107 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:26.945073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:26.945045 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6"] Oct 30 07:26:26.945583 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:26.945557 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67765ca0_2d32_43ad_a0ae_cab37e23dffc.slice/crio-b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66 WatchSource:0}: Error finding container b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66: Status 404 returned error can't find the container with id b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66 Oct 30 07:26:27.618485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:27.618394 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" event={"ID":"67765ca0-2d32-43ad-a0ae-cab37e23dffc","Type":"ContainerStarted","Data":"5ce48126d217a0eafa604f6343fec0ff7d3edaae51e5b625880b967d652da8b2"} Oct 30 07:26:27.618485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:27.618453 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" event={"ID":"67765ca0-2d32-43ad-a0ae-cab37e23dffc","Type":"ContainerStarted","Data":"b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66"} Oct 30 07:26:27.662512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:27.662455 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" podStartSLOduration=1.662440772 podStartE2EDuration="1.662440772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:27.659919242 +0000 UTC m=+1751.350836679" watchObservedRunningTime="2026-10-30 07:26:27.662440772 +0000 UTC m=+1751.353358208" Oct 30 07:26:27.725417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:27.725379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:27.996594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:27.996565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:28.626504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:28.626471 5894 generic.go:356] "Generic (PLEG): container finished" podID="67765ca0-2d32-43ad-a0ae-cab37e23dffc" containerID="5ce48126d217a0eafa604f6343fec0ff7d3edaae51e5b625880b967d652da8b2" exitCode=0 Oct 30 07:26:28.626951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:28.626511 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" event={"ID":"67765ca0-2d32-43ad-a0ae-cab37e23dffc","Type":"ContainerDied","Data":"5ce48126d217a0eafa604f6343fec0ff7d3edaae51e5b625880b967d652da8b2"} Oct 30 07:26:28.848855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:28.848793 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:26:28.926018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:28.925932 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:29.334872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.334835 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:29.799406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.799377 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:29.904578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.904501 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\") pod \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " Oct 30 07:26:29.904762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.904668 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\") pod \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " Oct 30 07:26:29.904816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.904800 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\" (UniqueName: \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\") pod \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\" (UID: \"67765ca0-2d32-43ad-a0ae-cab37e23dffc\") " Oct 30 07:26:29.905052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.905027 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca" pod "67765ca0-2d32-43ad-a0ae-cab37e23dffc" (UID: "67765ca0-2d32-43ad-a0ae-cab37e23dffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:29.905339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.905320 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:29.906808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.906784 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca" pod "67765ca0-2d32-43ad-a0ae-cab37e23dffc" (UID: "67765ca0-2d32-43ad-a0ae-cab37e23dffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:29.907106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.907085 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km" pod "67765ca0-2d32-43ad-a0ae-cab37e23dffc" (UID: "67765ca0-2d32-43ad-a0ae-cab37e23dffc"). InnerVolumeSpecName "kube-api-access-7h7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:29.945425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:29.945393 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:26:30.006528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006494 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.006759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006580 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.006759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006736 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.006924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006830 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.006924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006875 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.007033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.006954 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.007033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.007016 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.007149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.007088 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.007149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.007125 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\") pod \"210419a8-cabf-4139-b373-921abcf0141d\" (UID: \"210419a8-cabf-4139-b373-921abcf0141d\") " Oct 30 07:26:30.007873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.007844 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h7km\" (UniqueName: \"kubernetes.io/projected/67765ca0-2d32-43ad-a0ae-cab37e23dffc-kube-api-access-7h7km\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.007873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.007875 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67765ca0-2d32-43ad-a0ae-cab37e23dffc-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.008154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.008133 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:30.008474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.008323 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:30.008716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.008683 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:30.008985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.008961 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:30.009214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.009194 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:30.009335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.009208 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "kube-api-access-ln7bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:30.011185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.011158 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:30.011481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.011457 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:30.012202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.012167 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token" pod "210419a8-cabf-4139-b373-921abcf0141d" (UID: "210419a8-cabf-4139-b373-921abcf0141d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:30.109438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109409 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln7bc\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-kube-api-access-ln7bc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109436 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109447 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109457 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109465 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109473 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210419a8-cabf-4139-b373-921abcf0141d-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109486 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210419a8-cabf-4139-b373-921abcf0141d-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109499 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210419a8-cabf-4139-b373-921abcf0141d-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.109716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.109506 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210419a8-cabf-4139-b373-921abcf0141d-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:30.640856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.640822 5894 generic.go:356] "Generic (PLEG): container finished" podID="210419a8-cabf-4139-b373-921abcf0141d" containerID="90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387" exitCode=0 Oct 30 07:26:30.641040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.640888 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" Oct 30 07:26:30.641040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.640896 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" event={"ID":"210419a8-cabf-4139-b373-921abcf0141d","Type":"ContainerDied","Data":"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387"} Oct 30 07:26:30.641040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.640943 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs" event={"ID":"210419a8-cabf-4139-b373-921abcf0141d","Type":"ContainerDied","Data":"7db1a34f2e23d6392945a17f5920e015aabcaec41fa27f99b1a377e05306ffb8"} Oct 30 07:26:30.641040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.640960 5894 scope.go:122] "RemoveContainer" containerID="90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387" Oct 30 07:26:30.642570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.642534 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" Oct 30 07:26:30.642711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.642585 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6" event={"ID":"67765ca0-2d32-43ad-a0ae-cab37e23dffc","Type":"ContainerDied","Data":"b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66"} Oct 30 07:26:30.642711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.642663 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b73617cf72a7fba43ffd3b0081dfb520a01991eeebfe5f7032cbdd05febe66" Oct 30 07:26:30.659704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.659684 5894 scope.go:122] "RemoveContainer" containerID="90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387" Oct 30 07:26:30.660003 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:26:30.659985 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387\": container with ID starting with 90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387 not found: ID does not exist" containerID="90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387" Oct 30 07:26:30.660059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.660013 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387"} err="failed to get container status \"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387\": rpc error: code = NotFound desc = could not find container \"90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387\": container with ID starting with 90a026e3bec229571313aad920499a9f8b911e151b217aa7b0aebadfc4c5a387 not found: ID does not exist" Oct 30 07:26:30.710542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.710467 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:26:30.722651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:30.722612 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-58zvs"] Oct 30 07:26:31.142908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.142879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:31.685939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.685904 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr"] Oct 30 07:26:31.686440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686426 5894 cpu_manager.go:414] "containerMap: removing container" podUID="67765ca0-2d32-43ad-a0ae-cab37e23dffc" containerName="revoke-keys" Oct 30 07:26:31.686503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686441 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67765ca0-2d32-43ad-a0ae-cab37e23dffc" containerName="revoke-keys" Oct 30 07:26:31.686503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686465 5894 cpu_manager.go:414] "containerMap: removing container" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" Oct 30 07:26:31.686503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686470 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" Oct 30 07:26:31.686619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686576 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="210419a8-cabf-4139-b373-921abcf0141d" containerName="istio-proxy" Oct 30 07:26:31.686619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.686592 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="67765ca0-2d32-43ad-a0ae-cab37e23dffc" containerName="revoke-keys" Oct 30 07:26:31.691384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.691365 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.707492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.707467 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:31.710529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.710496 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr"] Oct 30 07:26:31.718925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.718867 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:31.718925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.718880 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:31.718925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.718908 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:31.730728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.730694 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.730867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.730764 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.730951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.730891 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dts5q\" (UniqueName: \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.832597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.832559 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.832810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.832654 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.832810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.832700 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dts5q\" (UniqueName: \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.833235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.833215 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.835020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.835001 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:31.844650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.844616 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:26:31.844857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.844833 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" containerID="cri-o://83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1" gracePeriod=30 Oct 30 07:26:31.862007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:31.861981 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dts5q\" (UniqueName: \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\") pod \"maas-api-revoke-sub-240cf2df-xnbpr\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:32.005335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:32.005256 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:32.309407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:32.309377 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr"] Oct 30 07:26:32.309872 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:32.309751 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e69efc_3c70_4a68_987e_4f8112f3b097.slice/crio-7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11 WatchSource:0}: Error finding container 7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11: Status 404 returned error can't find the container with id 7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11 Oct 30 07:26:32.432046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:32.432010 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210419a8-cabf-4139-b373-921abcf0141d" path="/var/lib/kubelet/pods/210419a8-cabf-4139-b373-921abcf0141d/volumes" Oct 30 07:26:32.660192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:32.660111 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" event={"ID":"95e69efc-3c70-4a68-987e-4f8112f3b097","Type":"ContainerStarted","Data":"7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11"} Oct 30 07:26:32.820302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:32.820266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:33.668058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:33.668018 5894 generic.go:356] "Generic (PLEG): container finished" podID="95e69efc-3c70-4a68-987e-4f8112f3b097" containerID="75ae9bd6ff5166457de75f6415b00832376a2a1c4c83fac487da56563362c722" exitCode=0 Oct 30 07:26:33.668520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:33.668106 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" event={"ID":"95e69efc-3c70-4a68-987e-4f8112f3b097","Type":"ContainerDied","Data":"75ae9bd6ff5166457de75f6415b00832376a2a1c4c83fac487da56563362c722"} Oct 30 07:26:34.442087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.442047 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:34.829579 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:26:34.829546 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:26:34.856904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.856873 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:34.970109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.970078 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\") pod \"95e69efc-3c70-4a68-987e-4f8112f3b097\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " Oct 30 07:26:34.970290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.970168 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\" (UniqueName: \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\") pod \"95e69efc-3c70-4a68-987e-4f8112f3b097\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " Oct 30 07:26:34.970370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.970353 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\") pod \"95e69efc-3c70-4a68-987e-4f8112f3b097\" (UID: \"95e69efc-3c70-4a68-987e-4f8112f3b097\") " Oct 30 07:26:34.970787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.970757 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca" pod "95e69efc-3c70-4a68-987e-4f8112f3b097" (UID: "95e69efc-3c70-4a68-987e-4f8112f3b097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:34.970919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.970865 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:34.972380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.972352 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q" pod "95e69efc-3c70-4a68-987e-4f8112f3b097" (UID: "95e69efc-3c70-4a68-987e-4f8112f3b097"). InnerVolumeSpecName "kube-api-access-dts5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:34.972479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:34.972404 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca" pod "95e69efc-3c70-4a68-987e-4f8112f3b097" (UID: "95e69efc-3c70-4a68-987e-4f8112f3b097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:35.072352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:35.072313 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dts5q\" (UniqueName: \"kubernetes.io/projected/95e69efc-3c70-4a68-987e-4f8112f3b097-kube-api-access-dts5q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:35.072352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:35.072350 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95e69efc-3c70-4a68-987e-4f8112f3b097-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:35.683189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:35.683159 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" Oct 30 07:26:35.683377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:35.683192 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr" event={"ID":"95e69efc-3c70-4a68-987e-4f8112f3b097","Type":"ContainerDied","Data":"7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11"} Oct 30 07:26:35.683377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:35.683240 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f1ea913d68d00a552c3854e33a84286556dca6d30447f33d4509eaaef036e11" Oct 30 07:26:36.237909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:36.237873 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:36.746981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:36.746947 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:26:36.747165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:36.747003 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:26:37.122510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.122481 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:26:37.200799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.200768 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.200955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.200815 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.200955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.200865 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.200955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.200928 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.200974 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201022 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201073 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201220 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201269 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\") pod \"55276e15-4d9c-42dd-ae4d-95859b544e27\" (UID: \"55276e15-4d9c-42dd-ae4d-95859b544e27\") " Oct 30 07:26:37.201458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201417 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:37.201960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201552 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:37.201960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201802 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:37.201960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201898 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.201960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201920 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.201960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.201935 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.202314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.202124 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:37.202314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.202203 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:37.203466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.203438 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:37.203627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.203607 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:26:37.203755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.203678 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:37.203755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.203707 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2" pod "55276e15-4d9c-42dd-ae4d-95859b544e27" (UID: "55276e15-4d9c-42dd-ae4d-95859b544e27"). InnerVolumeSpecName "kube-api-access-5ktr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:37.303386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303355 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.303386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303385 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55276e15-4d9c-42dd-ae4d-95859b544e27-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.303386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303396 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.303864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303406 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ktr2\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-kube-api-access-5ktr2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.303864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303418 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.303864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.303426 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55276e15-4d9c-42dd-ae4d-95859b544e27-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:37.698288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.698201 5894 generic.go:356] "Generic (PLEG): container finished" podID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerID="83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1" exitCode=0 Oct 30 07:26:37.698288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.698270 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" Oct 30 07:26:37.698529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.698281 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" event={"ID":"55276e15-4d9c-42dd-ae4d-95859b544e27","Type":"ContainerDied","Data":"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1"} Oct 30 07:26:37.698529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.698346 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh" event={"ID":"55276e15-4d9c-42dd-ae4d-95859b544e27","Type":"ContainerDied","Data":"d616b10b406cf0169af76e0d9d71fce2540a1095d87d946792b283c3ee9de2a9"} Oct 30 07:26:37.698529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.698376 5894 scope.go:122] "RemoveContainer" containerID="83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1" Oct 30 07:26:37.709751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.709707 5894 scope.go:122] "RemoveContainer" containerID="83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1" Oct 30 07:26:37.710059 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:26:37.710038 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1\": container with ID starting with 83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1 not found: ID does not exist" containerID="83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1" Oct 30 07:26:37.710126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.710069 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1"} err="failed to get container status \"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1\": rpc error: code = NotFound desc = could not find container \"83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1\": container with ID starting with 83dcc5de03512ff59fa97bc7f82bd8799096a77800622f3cb73c1467252781c1 not found: ID does not exist" Oct 30 07:26:37.763762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.763728 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:26:37.776347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.776311 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-4hhvh"] Oct 30 07:26:37.914341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:37.914303 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:38.050584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:38.050544 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:38.431605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:38.431512 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" path="/var/lib/kubelet/pods/55276e15-4d9c-42dd-ae4d-95859b544e27/volumes" Oct 30 07:26:39.084707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.084627 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls"] Oct 30 07:26:39.085435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085405 5894 cpu_manager.go:414] "containerMap: removing container" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" Oct 30 07:26:39.085435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085437 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" Oct 30 07:26:39.085605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085472 5894 cpu_manager.go:414] "containerMap: removing container" podUID="95e69efc-3c70-4a68-987e-4f8112f3b097" containerName="revoke-keys" Oct 30 07:26:39.085605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085480 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95e69efc-3c70-4a68-987e-4f8112f3b097" containerName="revoke-keys" Oct 30 07:26:39.085741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085724 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="55276e15-4d9c-42dd-ae4d-95859b544e27" containerName="istio-proxy" Oct 30 07:26:39.085780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.085751 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="95e69efc-3c70-4a68-987e-4f8112f3b097" containerName="revoke-keys" Oct 30 07:26:39.096683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.091009 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.124854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.119706 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:39.124854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.119755 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:39.124854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.119774 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:39.124854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.122439 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:39.131084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.131031 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls"] Oct 30 07:26:39.230621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.230583 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2djg\" (UniqueName: \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.230869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.230741 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.230869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.230773 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.332494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.332458 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2djg\" (UniqueName: \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.332687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.332554 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.332687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.332589 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.333278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.333257 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.335015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.334952 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.356980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.356952 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2djg\" (UniqueName: \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\") pod \"maas-api-revoke-sub-8774e7c6-d4cls\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.444455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.444418 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:39.721981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.721904 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:39.750457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:39.750427 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls"] Oct 30 07:26:39.750671 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:39.750625 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f53cf7d_e409_45ce_89a4_ff11ff2c4568.slice/crio-f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022 WatchSource:0}: Error finding container f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022: Status 404 returned error can't find the container with id f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022 Oct 30 07:26:40.729955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:40.729898 5894 generic.go:356] "Generic (PLEG): container finished" podID="8f53cf7d-e409-45ce-89a4-ff11ff2c4568" containerID="cec2cb4a2951be14722ab15926289cd6746ead542ab08dcb84ac3f89607d3f7b" exitCode=0 Oct 30 07:26:40.730358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:40.729974 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" event={"ID":"8f53cf7d-e409-45ce-89a4-ff11ff2c4568","Type":"ContainerDied","Data":"cec2cb4a2951be14722ab15926289cd6746ead542ab08dcb84ac3f89607d3f7b"} Oct 30 07:26:40.730358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:40.730022 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" event={"ID":"8f53cf7d-e409-45ce-89a4-ff11ff2c4568","Type":"ContainerStarted","Data":"f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022"} Oct 30 07:26:41.423159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.423125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:41.893230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.893201 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:41.967246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.967194 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\") pod \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " Oct 30 07:26:41.967470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.967262 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\" (UniqueName: \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\") pod \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " Oct 30 07:26:41.967470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.967386 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\") pod \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\" (UID: \"8f53cf7d-e409-45ce-89a4-ff11ff2c4568\") " Oct 30 07:26:41.967766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.967738 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca" pod "8f53cf7d-e409-45ce-89a4-ff11ff2c4568" (UID: "8f53cf7d-e409-45ce-89a4-ff11ff2c4568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:41.967965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.967943 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:41.969959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.969934 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca" pod "8f53cf7d-e409-45ce-89a4-ff11ff2c4568" (UID: "8f53cf7d-e409-45ce-89a4-ff11ff2c4568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:41.970070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:41.969963 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg" pod "8f53cf7d-e409-45ce-89a4-ff11ff2c4568" (UID: "8f53cf7d-e409-45ce-89a4-ff11ff2c4568"). InnerVolumeSpecName "kube-api-access-c2djg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:42.068832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:42.068799 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:42.068832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:42.068828 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2djg\" (UniqueName: \"kubernetes.io/projected/8f53cf7d-e409-45ce-89a4-ff11ff2c4568-kube-api-access-c2djg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:42.744873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:42.744785 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" event={"ID":"8f53cf7d-e409-45ce-89a4-ff11ff2c4568","Type":"ContainerDied","Data":"f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022"} Oct 30 07:26:42.744873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:42.744847 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7156fa366359a7b5e8c45c6ba6c186fd416429fe1321f77a9672d4e3d927022" Oct 30 07:26:42.744873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:42.744800 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls" Oct 30 07:26:43.519267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:43.519225 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:43.870799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:43.870762 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" podUID="97b32b94-135f-4181-94e2-be0ec3213b5c" containerName="main" containerID="cri-o://a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a" gracePeriod=45 Oct 30 07:26:44.137934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.137870 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:26:44.298124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298092 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\" (UniqueName: \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298167 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298189 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298217 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298259 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298389 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\") pod \"97b32b94-135f-4181-94e2-be0ec3213b5c\" (UID: \"97b32b94-135f-4181-94e2-be0ec3213b5c\") " Oct 30 07:26:44.298523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298506 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:44.298654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:44.298713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298648 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:44.298981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.298957 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.299112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.299006 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.299112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.299021 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.300569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.300544 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:26:44.300719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.300565 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "kube-api-access-rvj77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:44.300719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.300622 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs" pod "97b32b94-135f-4181-94e2-be0ec3213b5c" (UID: "97b32b94-135f-4181-94e2-be0ec3213b5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:44.400290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.400203 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvj77\" (UniqueName: \"kubernetes.io/projected/97b32b94-135f-4181-94e2-be0ec3213b5c-kube-api-access-rvj77\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.400290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.400234 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97b32b94-135f-4181-94e2-be0ec3213b5c-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.400290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.400247 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97b32b94-135f-4181-94e2-be0ec3213b5c-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:44.589368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.589327 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd"] Oct 30 07:26:44.590104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590081 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8f53cf7d-e409-45ce-89a4-ff11ff2c4568" containerName="revoke-keys" Oct 30 07:26:44.590204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590106 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f53cf7d-e409-45ce-89a4-ff11ff2c4568" containerName="revoke-keys" Oct 30 07:26:44.590204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590179 5894 cpu_manager.go:414] "containerMap: removing container" podUID="97b32b94-135f-4181-94e2-be0ec3213b5c" containerName="main" Oct 30 07:26:44.590204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590187 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b32b94-135f-4181-94e2-be0ec3213b5c" containerName="main" Oct 30 07:26:44.590353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590334 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b32b94-135f-4181-94e2-be0ec3213b5c" containerName="main" Oct 30 07:26:44.590402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.590355 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f53cf7d-e409-45ce-89a4-ff11ff2c4568" containerName="revoke-keys" Oct 30 07:26:44.593888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.593867 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.609657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.609612 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:44.612994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.612969 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:44.613106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.613083 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd"] Oct 30 07:26:44.622256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.622227 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:44.704345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.704255 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.704345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.704314 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctxbv\" (UniqueName: \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.704564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.704541 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.710783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.710756 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:44.760141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.760104 5894 generic.go:356] "Generic (PLEG): container finished" podID="97b32b94-135f-4181-94e2-be0ec3213b5c" containerID="a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a" exitCode=0 Oct 30 07:26:44.760319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.760151 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" Oct 30 07:26:44.760319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.760187 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" event={"ID":"97b32b94-135f-4181-94e2-be0ec3213b5c","Type":"ContainerDied","Data":"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a"} Oct 30 07:26:44.760319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.760233 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq" event={"ID":"97b32b94-135f-4181-94e2-be0ec3213b5c","Type":"ContainerDied","Data":"1d22a4b55b97ceaac350ad517f7cfa89cd66166637c9b06c37fd34ec6929a796"} Oct 30 07:26:44.760319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.760261 5894 scope.go:122] "RemoveContainer" containerID="a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a" Oct 30 07:26:44.773270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.773242 5894 scope.go:122] "RemoveContainer" containerID="a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a" Oct 30 07:26:44.773753 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:26:44.773618 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a\": container with ID starting with a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a not found: ID does not exist" containerID="a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a" Oct 30 07:26:44.773753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.773709 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a"} err="failed to get container status \"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a\": rpc error: code = NotFound desc = could not find container \"a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a\": container with ID starting with a9f0e107252e3b9a17fc7da0c2a3f70eb458aa75e3085ec64de49be2ff9e5b7a not found: ID does not exist" Oct 30 07:26:44.805874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.805841 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.806042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.805904 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctxbv\" (UniqueName: \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.806085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.806048 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.806680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.806654 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.808709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.808688 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.812446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.812417 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:26:44.816968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.816942 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-17bd11/ipp-route-17bd11-kserve-795c875557-vcsfq"] Oct 30 07:26:44.830111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.830086 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctxbv\" (UniqueName: \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\") pod \"maas-api-revoke-sub-96064fb5-r6wdd\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:44.907185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:44.907148 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:45.149500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:45.149471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:45.213662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:45.213561 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd"] Oct 30 07:26:45.214249 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:45.214223 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c3051c_c378_4b76_841e_124b2c88977a.slice/crio-876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40 WatchSource:0}: Error finding container 876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40: Status 404 returned error can't find the container with id 876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40 Oct 30 07:26:45.768867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:45.768774 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" event={"ID":"82c3051c-c378-4b76-841e-124b2c88977a","Type":"ContainerStarted","Data":"a47923d165e0c2fde02c690f972f69e2357fba9866d22996dbaead2cd35f18cc"} Oct 30 07:26:45.768867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:45.768855 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" event={"ID":"82c3051c-c378-4b76-841e-124b2c88977a","Type":"ContainerStarted","Data":"876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40"} Oct 30 07:26:46.431168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.431090 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b32b94-135f-4181-94e2-be0ec3213b5c" path="/var/lib/kubelet/pods/97b32b94-135f-4181-94e2-be0ec3213b5c/volumes" Oct 30 07:26:46.777390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.777362 5894 generic.go:356] "Generic (PLEG): container finished" podID="82c3051c-c378-4b76-841e-124b2c88977a" containerID="a47923d165e0c2fde02c690f972f69e2357fba9866d22996dbaead2cd35f18cc" exitCode=0 Oct 30 07:26:46.777849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.777453 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" event={"ID":"82c3051c-c378-4b76-841e-124b2c88977a","Type":"ContainerDied","Data":"a47923d165e0c2fde02c690f972f69e2357fba9866d22996dbaead2cd35f18cc"} Oct 30 07:26:46.851609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.851573 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:46.925349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.925316 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:26:46.929326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.929292 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:46.955128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:46.955086 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:26:47.034938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.034852 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.034938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.034918 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.034952 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.034969 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.035036 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.035130 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.035228 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.035261 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm7zd\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.035353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.035277 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.136944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.136905 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.136976 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137000 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137016 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137039 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137063 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137159 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137210 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm7zd\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137252 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137704 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137731 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137888 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.137996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.137967 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.138320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.138298 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.140029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.140008 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.140144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.140125 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.160645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.160595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.160817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.160793 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm7zd\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.243531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.243496 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:47.562844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.562813 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:26:47.563397 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:47.563371 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631af43a_06eb_4902_ab4f_18a0134d2dae.slice/crio-f96b185d1342247d52b90cf30b1b5fc45d96ebc7bad18380b791b458f72547e7 WatchSource:0}: Error finding container f96b185d1342247d52b90cf30b1b5fc45d96ebc7bad18380b791b458f72547e7: Status 404 returned error can't find the container with id f96b185d1342247d52b90cf30b1b5fc45d96ebc7bad18380b791b458f72547e7 Oct 30 07:26:47.566208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.566177 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.566305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.566247 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.566305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.566275 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:26:47.638593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.638562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:47.786972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.786926 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" event={"ID":"631af43a-06eb-4902-ab4f-18a0134d2dae","Type":"ContainerStarted","Data":"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a"} Oct 30 07:26:47.787439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.787003 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" event={"ID":"631af43a-06eb-4902-ab4f-18a0134d2dae","Type":"ContainerStarted","Data":"f96b185d1342247d52b90cf30b1b5fc45d96ebc7bad18380b791b458f72547e7"} Oct 30 07:26:47.833288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.833172 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podStartSLOduration=1.8331537980000001 podStartE2EDuration="1.833153798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:47.831698471 +0000 UTC m=+1771.522615909" watchObservedRunningTime="2026-10-30 07:26:47.833153798 +0000 UTC m=+1771.524071236" Oct 30 07:26:47.932313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.932287 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:47.946265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.946235 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\") pod \"82c3051c-c378-4b76-841e-124b2c88977a\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " Oct 30 07:26:47.946433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.946305 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\" (UniqueName: \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\") pod \"82c3051c-c378-4b76-841e-124b2c88977a\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " Oct 30 07:26:47.946433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.946402 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\") pod \"82c3051c-c378-4b76-841e-124b2c88977a\" (UID: \"82c3051c-c378-4b76-841e-124b2c88977a\") " Oct 30 07:26:47.946770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.946743 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca" pod "82c3051c-c378-4b76-841e-124b2c88977a" (UID: "82c3051c-c378-4b76-841e-124b2c88977a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:47.948392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.948363 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca" pod "82c3051c-c378-4b76-841e-124b2c88977a" (UID: "82c3051c-c378-4b76-841e-124b2c88977a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:47.948754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:47.948623 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv" pod "82c3051c-c378-4b76-841e-124b2c88977a" (UID: "82c3051c-c378-4b76-841e-124b2c88977a"). InnerVolumeSpecName "kube-api-access-ctxbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:48.047774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.047735 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:48.047774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.047766 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctxbv\" (UniqueName: \"kubernetes.io/projected/82c3051c-c378-4b76-841e-124b2c88977a-kube-api-access-ctxbv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:48.047774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.047776 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c3051c-c378-4b76-841e-124b2c88977a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:48.244287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.244255 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:48.245704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.245665 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:48.245848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.245724 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:48.498045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.498004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:48.804853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.804820 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" Oct 30 07:26:48.804853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.804841 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd" event={"ID":"82c3051c-c378-4b76-841e-124b2c88977a","Type":"ContainerDied","Data":"876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40"} Oct 30 07:26:48.805282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:48.804901 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876e025489d4cf07085aaec9e27c33a1ea3139a1ef13a7e8d98789692924fa40" Oct 30 07:26:49.244607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.244525 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:49.244607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.244580 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:49.521817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.521736 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj"] Oct 30 07:26:49.522340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.522325 5894 cpu_manager.go:414] "containerMap: removing container" podUID="82c3051c-c378-4b76-841e-124b2c88977a" containerName="revoke-keys" Oct 30 07:26:49.522379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.522341 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c3051c-c378-4b76-841e-124b2c88977a" containerName="revoke-keys" Oct 30 07:26:49.522459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.522449 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c3051c-c378-4b76-841e-124b2c88977a" containerName="revoke-keys" Oct 30 07:26:49.525990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.525968 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.554658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.554577 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:49.554658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.554615 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:49.554890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.554621 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:49.554890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.554585 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:49.557994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.557967 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj"] Oct 30 07:26:49.569955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.569908 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.579316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.578683 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78jc\" (UniqueName: \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.579316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.578951 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.682501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.682451 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.702916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.687623 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.728962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.703284 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.729154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.729110 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78jc\" (UniqueName: \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.740666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.732350 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.754666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.753793 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78jc\" (UniqueName: \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\") pod \"maas-api-revoke-sub-4cb83fbe-tc4dj\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:49.840246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:49.840208 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:50.207053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.206923 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj"] Oct 30 07:26:50.211397 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:50.211353 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe64f4f_1a8f_459b_937c_745dd582d680.slice/crio-1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97 WatchSource:0}: Error finding container 1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97: Status 404 returned error can't find the container with id 1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97 Oct 30 07:26:50.244716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.244689 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:50.244838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.244755 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:50.419358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.419237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:50.839683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.837515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" event={"ID":"3fe64f4f-1a8f-459b-937c-745dd582d680","Type":"ContainerStarted","Data":"b1b863a1af42f1994b1bd6e9ecbcd6c7673a9bd2a5db8d479b8cbfbc97aee24a"} Oct 30 07:26:50.839683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.837954 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" event={"ID":"3fe64f4f-1a8f-459b-937c-745dd582d680","Type":"ContainerStarted","Data":"1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97"} Oct 30 07:26:50.914448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:50.914317 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" podStartSLOduration=1.914296139 podStartE2EDuration="1.914296139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:50.913906496 +0000 UTC m=+1774.604823936" watchObservedRunningTime="2026-10-30 07:26:50.914296139 +0000 UTC m=+1774.605213577" Oct 30 07:26:51.248716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:51.247816 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:51.248716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:51.247873 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:51.911598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:51.911065 5894 generic.go:356] "Generic (PLEG): container finished" podID="3fe64f4f-1a8f-459b-937c-745dd582d680" containerID="b1b863a1af42f1994b1bd6e9ecbcd6c7673a9bd2a5db8d479b8cbfbc97aee24a" exitCode=0 Oct 30 07:26:51.911598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:51.911206 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" event={"ID":"3fe64f4f-1a8f-459b-937c-745dd582d680","Type":"ContainerDied","Data":"b1b863a1af42f1994b1bd6e9ecbcd6c7673a9bd2a5db8d479b8cbfbc97aee24a"} Oct 30 07:26:52.256712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:52.255775 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:52.256712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:52.255835 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:52.803967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:52.803922 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:53.244120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.244088 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:53.244264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.244144 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:53.246990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.246965 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:53.384171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.381152 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\" (UniqueName: \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\") pod \"3fe64f4f-1a8f-459b-937c-745dd582d680\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " Oct 30 07:26:53.384171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.381302 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\") pod \"3fe64f4f-1a8f-459b-937c-745dd582d680\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " Oct 30 07:26:53.384171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.381434 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\") pod \"3fe64f4f-1a8f-459b-937c-745dd582d680\" (UID: \"3fe64f4f-1a8f-459b-937c-745dd582d680\") " Oct 30 07:26:53.388665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.388332 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca" pod "3fe64f4f-1a8f-459b-937c-745dd582d680" (UID: "3fe64f4f-1a8f-459b-937c-745dd582d680"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:53.400696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.400187 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc" pod "3fe64f4f-1a8f-459b-937c-745dd582d680" (UID: "3fe64f4f-1a8f-459b-937c-745dd582d680"). InnerVolumeSpecName "kube-api-access-g78jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:53.403650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.403603 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca" pod "3fe64f4f-1a8f-459b-937c-745dd582d680" (UID: "3fe64f4f-1a8f-459b-937c-745dd582d680"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:53.492663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.489912 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:53.492663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.489954 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fe64f4f-1a8f-459b-937c-745dd582d680-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:53.492663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.489970 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g78jc\" (UniqueName: \"kubernetes.io/projected/3fe64f4f-1a8f-459b-937c-745dd582d680-kube-api-access-g78jc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:53.955412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.955376 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" Oct 30 07:26:53.955412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.955390 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj" event={"ID":"3fe64f4f-1a8f-459b-937c-745dd582d680","Type":"ContainerDied","Data":"1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97"} Oct 30 07:26:53.955626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:53.955439 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1673c6e7898eefacbff8430dac2915972204550bea4380d38654b15a1a381c97" Oct 30 07:26:54.244353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.244266 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 07:26:54.244353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.244333 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 07:26:54.525832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.525747 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:54.944768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.941532 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k"] Oct 30 07:26:54.944768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.942418 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3fe64f4f-1a8f-459b-937c-745dd582d680" containerName="revoke-keys" Oct 30 07:26:54.944768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.942441 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fe64f4f-1a8f-459b-937c-745dd582d680" containerName="revoke-keys" Oct 30 07:26:54.944768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.942722 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fe64f4f-1a8f-459b-937c-745dd582d680" containerName="revoke-keys" Oct 30 07:26:54.946969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.946944 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:54.979954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.977879 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:26:54.979954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.977897 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:26:54.979954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.977935 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:26:54.979954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.977963 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:26:54.981728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:54.981700 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k"] Oct 30 07:26:55.124131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.124090 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.124131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.124129 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmnp\" (UniqueName: \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.124368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.124339 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.226444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.226352 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.226444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.226394 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmnp\" (UniqueName: \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.226733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.226519 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.227527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.227499 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.229760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.229730 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.251516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.251477 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmnp\" (UniqueName: \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\") pod \"maas-api-revoke-sub-4fb3ce34-sw24k\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.273202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.273164 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:55.583708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.583681 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k"] Oct 30 07:26:55.585730 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:26:55.585703 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c2632c_1988_45c7_9240_1dfdde0ddda5.slice/crio-adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb WatchSource:0}: Error finding container adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb: Status 404 returned error can't find the container with id adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb Oct 30 07:26:55.984686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.979377 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" event={"ID":"a5c2632c-1988-45c7-9240-1dfdde0ddda5","Type":"ContainerStarted","Data":"46a48f26267a91942fd7e65577d175ee6b6cee239187187179a2c6b72f55957d"} Oct 30 07:26:55.984686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:55.979448 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" event={"ID":"a5c2632c-1988-45c7-9240-1dfdde0ddda5","Type":"ContainerStarted","Data":"adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb"} Oct 30 07:26:56.020936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.020879 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" podStartSLOduration=2.020861279 podStartE2EDuration="2.020861279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:26:56.020199584 +0000 UTC m=+1779.711117021" watchObservedRunningTime="2026-10-30 07:26:56.020861279 +0000 UTC m=+1779.711778715" Oct 30 07:26:56.259264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.259169 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:56.295442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.295401 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:26:56.297906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.297850 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:26:56.490706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.490642 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:56.849586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.849539 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:56.850124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.850018 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:56.850974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.850952 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:26:56.989775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.989689 5894 generic.go:356] "Generic (PLEG): container finished" podID="a5c2632c-1988-45c7-9240-1dfdde0ddda5" containerID="46a48f26267a91942fd7e65577d175ee6b6cee239187187179a2c6b72f55957d" exitCode=0 Oct 30 07:26:56.989955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:56.989780 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" event={"ID":"a5c2632c-1988-45c7-9240-1dfdde0ddda5","Type":"ContainerDied","Data":"46a48f26267a91942fd7e65577d175ee6b6cee239187187179a2c6b72f55957d"} Oct 30 07:26:57.265242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:57.265163 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:58.164807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.164774 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:58.194272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.194242 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:26:58.276358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.276273 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\" (UniqueName: \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\") pod \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " Oct 30 07:26:58.276521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.276394 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\") pod \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " Oct 30 07:26:58.276658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.276623 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\") pod \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\" (UID: \"a5c2632c-1988-45c7-9240-1dfdde0ddda5\") " Oct 30 07:26:58.277095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.277061 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca" pod "a5c2632c-1988-45c7-9240-1dfdde0ddda5" (UID: "a5c2632c-1988-45c7-9240-1dfdde0ddda5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:26:58.277220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.277197 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:58.283888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.283845 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp" pod "a5c2632c-1988-45c7-9240-1dfdde0ddda5" (UID: "a5c2632c-1988-45c7-9240-1dfdde0ddda5"). InnerVolumeSpecName "kube-api-access-2gmnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:26:58.284135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.283942 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca" pod "a5c2632c-1988-45c7-9240-1dfdde0ddda5" (UID: "a5c2632c-1988-45c7-9240-1dfdde0ddda5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:26:58.378778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.378746 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gmnp\" (UniqueName: \"kubernetes.io/projected/a5c2632c-1988-45c7-9240-1dfdde0ddda5-kube-api-access-2gmnp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:58.378778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:58.378776 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5c2632c-1988-45c7-9240-1dfdde0ddda5-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:26:59.009020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.008985 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" Oct 30 07:26:59.009173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.009022 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k" event={"ID":"a5c2632c-1988-45c7-9240-1dfdde0ddda5","Type":"ContainerDied","Data":"adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb"} Oct 30 07:26:59.009173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.009098 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf04677e66c809330e1c7d4acd479c9284ce8d8d183e44ceea0c0d41cbecebb" Oct 30 07:26:59.327324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.327287 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:26:59.327825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.327499 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" containerID="cri-o://0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0" gracePeriod=30 Oct 30 07:26:59.983739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:26:59.983702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:00.632343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.632311 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:27:00.632946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.632929 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a5c2632c-1988-45c7-9240-1dfdde0ddda5" containerName="revoke-keys" Oct 30 07:27:00.632992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.632949 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c2632c-1988-45c7-9240-1dfdde0ddda5" containerName="revoke-keys" Oct 30 07:27:00.633104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.633094 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c2632c-1988-45c7-9240-1dfdde0ddda5" containerName="revoke-keys" Oct 30 07:27:00.636388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.636366 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.666980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.666939 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:27:00.707887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.707841 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.707992 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708043 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708137 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kzm4\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708181 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708286 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708318 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708374 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.708486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.708434 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809280 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809338 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kzm4\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809358 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809387 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809403 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809424 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809582 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809723 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809846 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.809869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809836 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.810090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.809998 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.810090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.810036 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.810202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.810159 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.810364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.810342 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.811772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.811752 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.811999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.811980 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.833294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.833261 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.833564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.833543 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kzm4\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\") pod \"e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:00.950173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:00.950089 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:01.272382 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:01.272343 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd724f182_c79e_4a68_951e_a09891b13f4c.slice/crio-bb84fb9251032fe53f0cdcb4c24d1450d7594cac67297c8fbf361526281277a4 WatchSource:0}: Error finding container bb84fb9251032fe53f0cdcb4c24d1450d7594cac67297c8fbf361526281277a4: Status 404 returned error can't find the container with id bb84fb9251032fe53f0cdcb4c24d1450d7594cac67297c8fbf361526281277a4 Oct 30 07:27:01.272578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:01.272555 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:27:01.274893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:01.274859 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:01.274997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:01.274952 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:01.275066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:01.274999 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:01.867530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:01.867498 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:02.043153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.043103 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" event={"ID":"d724f182-c79e-4a68-951e-a09891b13f4c","Type":"ContainerStarted","Data":"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e"} Oct 30 07:27:02.043335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.043172 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" event={"ID":"d724f182-c79e-4a68-951e-a09891b13f4c","Type":"ContainerStarted","Data":"bb84fb9251032fe53f0cdcb4c24d1450d7594cac67297c8fbf361526281277a4"} Oct 30 07:27:02.123210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.123114 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podStartSLOduration=2.12310049 podStartE2EDuration="2.12310049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:02.122613701 +0000 UTC m=+1785.813531139" watchObservedRunningTime="2026-10-30 07:27:02.12310049 +0000 UTC m=+1785.814017927" Oct 30 07:27:02.950833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.950797 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:02.952463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.952434 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:02.952548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:02.952483 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:03.218043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:03.217951 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:03.360774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:03.360733 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:03.629613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:03.629582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:03.950772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:03.950697 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:03.950772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:03.950752 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:04.230128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.230051 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk"] Oct 30 07:27:04.234071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.234043 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.263124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.263095 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:04.263298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.263152 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:04.263677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.263654 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.263756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.263703 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.263878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.263860 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksjh\" (UniqueName: \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.267056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.267031 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:04.267814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.267789 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk"] Oct 30 07:27:04.280281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.280249 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:04.293970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.293930 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:04.368901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.365765 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksjh\" (UniqueName: \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.368901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.366309 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.368901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.366355 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.368901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.367111 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.369967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.369940 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.402661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.402502 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksjh\" (UniqueName: \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\") pod \"maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.509520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.509436 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688"] Oct 30 07:27:04.513784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.513762 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.546930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.540773 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688"] Oct 30 07:27:04.549411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.549382 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:04.595658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.569232 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhpr4\" (UniqueName: \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.595658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.569335 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.595658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.569461 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.654914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.654885 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:27:04.694908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.670413 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.694908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.670674 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.694908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.670729 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhpr4\" (UniqueName: \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.694908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.671834 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.713434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.713003 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.716252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.716219 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhpr4\" (UniqueName: \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\") pod \"maas-api-revoke-sub-9444437c-5w688\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777081 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777259 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777387 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777425 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777493 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777531 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777573 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.792722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.777613 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\") pod \"1abb1e21-3df2-4cd9-af05-870e82de81d3\" (UID: \"1abb1e21-3df2-4cd9-af05-870e82de81d3\") " Oct 30 07:27:04.804545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.804502 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:04.804922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.804749 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:04.805475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.805205 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:04.805475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.805410 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:04.805801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.805583 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:04.807934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.807382 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:04.810239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.810210 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "kube-api-access-v9bgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:04.810740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.810284 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:27:04.814988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.814908 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy" pod "1abb1e21-3df2-4cd9-af05-870e82de81d3" (UID: "1abb1e21-3df2-4cd9-af05-870e82de81d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:27:04.839996 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:27:04.839953 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:27:04.840246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.840231 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.895686 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bgn\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-kube-api-access-v9bgn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896737 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896768 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896783 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896797 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896811 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1abb1e21-3df2-4cd9-af05-870e82de81d3-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896828 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896841 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.896912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.896854 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1abb1e21-3df2-4cd9-af05-870e82de81d3-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:04.917713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.911767 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk"] Oct 30 07:27:04.917713 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:04.911770 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173c5454_3ebf_40e9_9917_454b0f5e5a46.slice/crio-60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870 WatchSource:0}: Error finding container 60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870: Status 404 returned error can't find the container with id 60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870 Oct 30 07:27:04.954836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.954802 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:04.955244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:04.954885 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:05.068525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.068325 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" event={"ID":"173c5454-3ebf-40e9-9917-454b0f5e5a46","Type":"ContainerStarted","Data":"60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870"} Oct 30 07:27:05.070299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.070264 5894 generic.go:356] "Generic (PLEG): container finished" podID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerID="0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0" exitCode=0 Oct 30 07:27:05.070448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.070348 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" event={"ID":"1abb1e21-3df2-4cd9-af05-870e82de81d3","Type":"ContainerDied","Data":"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0"} Oct 30 07:27:05.070448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.070392 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" event={"ID":"1abb1e21-3df2-4cd9-af05-870e82de81d3","Type":"ContainerDied","Data":"e49a45467e3329623ba36911aac77c71fed0a45159c052d851f53c2dc9a229a5"} Oct 30 07:27:05.070448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.070421 5894 scope.go:122] "RemoveContainer" containerID="0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0" Oct 30 07:27:05.070619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.070595 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn" Oct 30 07:27:05.114899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.113411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:05.153069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.153042 5894 scope.go:122] "RemoveContainer" containerID="0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0" Oct 30 07:27:05.153468 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:27:05.153448 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0\": container with ID starting with 0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0 not found: ID does not exist" containerID="0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0" Oct 30 07:27:05.153693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.153671 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0"} err="failed to get container status \"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0\": rpc error: code = NotFound desc = could not find container \"0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0\": container with ID starting with 0437b3a5119d8f1e78ef3cf209045af979faf3d38d37863d05170df900ffc1b0 not found: ID does not exist" Oct 30 07:27:05.217599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.217541 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:27:05.227662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.221272 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-k7lsn"] Oct 30 07:27:05.227662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.223617 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688"] Oct 30 07:27:05.227662 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:05.226487 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f14311_27f1_48c5_8e78_6bc3cd9f3caf.slice/crio-3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949 WatchSource:0}: Error finding container 3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949: Status 404 returned error can't find the container with id 3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949 Oct 30 07:27:05.950883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.950848 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:05.951037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:05.950900 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:06.108235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.108196 5894 generic.go:356] "Generic (PLEG): container finished" podID="173c5454-3ebf-40e9-9917-454b0f5e5a46" containerID="2354ce7758166dc64741c57f6f60eab2db6720f419ae0d1f2554a9386fdaedd3" exitCode=0 Oct 30 07:27:06.108751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.108342 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" event={"ID":"173c5454-3ebf-40e9-9917-454b0f5e5a46","Type":"ContainerDied","Data":"2354ce7758166dc64741c57f6f60eab2db6720f419ae0d1f2554a9386fdaedd3"} Oct 30 07:27:06.110730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.110696 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" event={"ID":"40f14311-27f1-48c5-8e78-6bc3cd9f3caf","Type":"ContainerStarted","Data":"af247ef5b449f4528d8be8c93208e0b170b25d443e65ef3cbaf17d5773c1311a"} Oct 30 07:27:06.110830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.110761 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" event={"ID":"40f14311-27f1-48c5-8e78-6bc3cd9f3caf","Type":"ContainerStarted","Data":"3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949"} Oct 30 07:27:06.195749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.195679 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" podStartSLOduration=2.195657456 podStartE2EDuration="2.195657456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:06.194076395 +0000 UTC m=+1789.884993835" watchObservedRunningTime="2026-10-30 07:27:06.195657456 +0000 UTC m=+1789.886574894" Oct 30 07:27:06.434801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.434769 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" path="/var/lib/kubelet/pods/1abb1e21-3df2-4cd9-af05-870e82de81d3/volumes" Oct 30 07:27:06.534364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.534324 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:06.951001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.950967 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:06.951168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:06.951023 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:07.121950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.121916 5894 generic.go:356] "Generic (PLEG): container finished" podID="40f14311-27f1-48c5-8e78-6bc3cd9f3caf" containerID="af247ef5b449f4528d8be8c93208e0b170b25d443e65ef3cbaf17d5773c1311a" exitCode=0 Oct 30 07:27:07.122405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.122121 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" event={"ID":"40f14311-27f1-48c5-8e78-6bc3cd9f3caf","Type":"ContainerDied","Data":"af247ef5b449f4528d8be8c93208e0b170b25d443e65ef3cbaf17d5773c1311a"} Oct 30 07:27:07.346073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.346047 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:07.448213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.448175 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\") pod \"173c5454-3ebf-40e9-9917-454b0f5e5a46\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " Oct 30 07:27:07.448427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.448255 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\" (UniqueName: \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\") pod \"173c5454-3ebf-40e9-9917-454b0f5e5a46\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " Oct 30 07:27:07.448427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.448421 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\") pod \"173c5454-3ebf-40e9-9917-454b0f5e5a46\" (UID: \"173c5454-3ebf-40e9-9917-454b0f5e5a46\") " Oct 30 07:27:07.449029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.448991 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca" pod "173c5454-3ebf-40e9-9917-454b0f5e5a46" (UID: "173c5454-3ebf-40e9-9917-454b0f5e5a46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:07.450808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.450779 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh" pod "173c5454-3ebf-40e9-9917-454b0f5e5a46" (UID: "173c5454-3ebf-40e9-9917-454b0f5e5a46"). InnerVolumeSpecName "kube-api-access-cksjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:07.450918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.450845 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca" pod "173c5454-3ebf-40e9-9917-454b0f5e5a46" (UID: "173c5454-3ebf-40e9-9917-454b0f5e5a46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:07.550659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.550582 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:07.550659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.550648 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cksjh\" (UniqueName: \"kubernetes.io/projected/173c5454-3ebf-40e9-9917-454b0f5e5a46-kube-api-access-cksjh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:07.550902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.550667 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/173c5454-3ebf-40e9-9917-454b0f5e5a46-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:07.951015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.950926 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 07:27:07.951015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:07.950984 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 07:27:08.132292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.132258 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" Oct 30 07:27:08.132806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.132303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk" event={"ID":"173c5454-3ebf-40e9-9917-454b0f5e5a46","Type":"ContainerDied","Data":"60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870"} Oct 30 07:27:08.132806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.132352 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e9117e1d4294f15ab34208b7b4bc9cef9d0d113f81932f87b4964ede931870" Oct 30 07:27:08.172886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.172843 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x"] Oct 30 07:27:08.173446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173428 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" Oct 30 07:27:08.173446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173443 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" Oct 30 07:27:08.173675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173474 5894 cpu_manager.go:414] "containerMap: removing container" podUID="173c5454-3ebf-40e9-9917-454b0f5e5a46" containerName="revoke-keys" Oct 30 07:27:08.173675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173481 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="173c5454-3ebf-40e9-9917-454b0f5e5a46" containerName="revoke-keys" Oct 30 07:27:08.173675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173584 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abb1e21-3df2-4cd9-af05-870e82de81d3" containerName="istio-proxy" Oct 30 07:27:08.173675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.173598 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="173c5454-3ebf-40e9-9917-454b0f5e5a46" containerName="revoke-keys" Oct 30 07:27:08.178912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.178889 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.204224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.199779 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x"] Oct 30 07:27:08.249680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.249578 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:08.260388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.260357 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpnq5\" (UniqueName: \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.260530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.260420 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.260602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.260590 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.342416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.342380 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:08.361865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.361829 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpnq5\" (UniqueName: \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.362037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.361907 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.362037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.362007 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.362767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.362741 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.364484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.364454 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.386152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.386119 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpnq5\" (UniqueName: \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\") pod \"maas-api-revoke-sub-3ee30481-pfk8x\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.463084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.463001 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\") pod \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " Oct 30 07:27:08.463245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.463230 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\") pod \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " Oct 30 07:27:08.463315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.463302 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\" (UniqueName: \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\") pod \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\" (UID: \"40f14311-27f1-48c5-8e78-6bc3cd9f3caf\") " Oct 30 07:27:08.463565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.463529 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca" pod "40f14311-27f1-48c5-8e78-6bc3cd9f3caf" (UID: "40f14311-27f1-48c5-8e78-6bc3cd9f3caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:08.468113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.468080 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4" pod "40f14311-27f1-48c5-8e78-6bc3cd9f3caf" (UID: "40f14311-27f1-48c5-8e78-6bc3cd9f3caf"). InnerVolumeSpecName "kube-api-access-nhpr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:08.470575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.470548 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh"] Oct 30 07:27:08.471510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.471481 5894 cpu_manager.go:414] "containerMap: removing container" podUID="40f14311-27f1-48c5-8e78-6bc3cd9f3caf" containerName="revoke-keys" Oct 30 07:27:08.471510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.471509 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f14311-27f1-48c5-8e78-6bc3cd9f3caf" containerName="revoke-keys" Oct 30 07:27:08.471739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.471689 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f14311-27f1-48c5-8e78-6bc3cd9f3caf" containerName="revoke-keys" Oct 30 07:27:08.474183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.474156 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca" pod "40f14311-27f1-48c5-8e78-6bc3cd9f3caf" (UID: "40f14311-27f1-48c5-8e78-6bc3cd9f3caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:08.475927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.475905 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.497446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.497414 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:08.498933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.497828 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh"] Oct 30 07:27:08.567339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567285 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.567534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567389 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpx5f\" (UniqueName: \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.567534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567506 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.567697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567611 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:08.567697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567629 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhpr4\" (UniqueName: \"kubernetes.io/projected/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-kube-api-access-nhpr4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:08.567697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.567661 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f14311-27f1-48c5-8e78-6bc3cd9f3caf-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:08.668729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.668695 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.668918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.668769 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx5f\" (UniqueName: \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.668918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.668851 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.669493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.669469 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.671284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.671258 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.692534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.692497 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx5f\" (UniqueName: \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\") pod \"maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.821479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.821444 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:08.826421 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:08.826387 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a2911a_62aa_4753_a002_156ef9013c14.slice/crio-1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e WatchSource:0}: Error finding container 1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e: Status 404 returned error can't find the container with id 1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e Oct 30 07:27:08.828270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:08.828240 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x"] Oct 30 07:27:09.033742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.033707 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:09.034174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.034139 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:09.035140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.035116 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:27:09.141626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.141549 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" Oct 30 07:27:09.141626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.141591 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688" event={"ID":"40f14311-27f1-48c5-8e78-6bc3cd9f3caf","Type":"ContainerDied","Data":"3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949"} Oct 30 07:27:09.141626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.141662 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e8ab64e2998c95a0f2876f19ee51fc08162fa740d76a714c152d6752e1b6949" Oct 30 07:27:09.143124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.143099 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" event={"ID":"a4a2911a-62aa-4753-a002-156ef9013c14","Type":"ContainerStarted","Data":"1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e"} Oct 30 07:27:09.215128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:09.215088 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh"] Oct 30 07:27:10.153213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.153134 5894 generic.go:356] "Generic (PLEG): container finished" podID="a4a2911a-62aa-4753-a002-156ef9013c14" containerID="1585b220ac41bcbd0c1993e11d3891a2e1b0d9db122b5728aaf261d94db3cef5" exitCode=0 Oct 30 07:27:10.153595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.153224 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" event={"ID":"a4a2911a-62aa-4753-a002-156ef9013c14","Type":"ContainerDied","Data":"1585b220ac41bcbd0c1993e11d3891a2e1b0d9db122b5728aaf261d94db3cef5"} Oct 30 07:27:10.155028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.154996 5894 generic.go:356] "Generic (PLEG): container finished" podID="4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" containerID="f172b5b87b932a1b893e26b94fb880367872ec1a6bedce43b2991f74ca4edaa3" exitCode=0 Oct 30 07:27:10.155167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.155072 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" event={"ID":"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb","Type":"ContainerDied","Data":"f172b5b87b932a1b893e26b94fb880367872ec1a6bedce43b2991f74ca4edaa3"} Oct 30 07:27:10.155167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.155114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" event={"ID":"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb","Type":"ContainerStarted","Data":"e40796249f8134c7ae03c31df04c23647e77e1959859e0d4d398dc52599afaf1"} Oct 30 07:27:10.196517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:10.196483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:11.355921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.355895 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:11.359627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.359606 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:11.405013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.404984 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\") pod \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " Oct 30 07:27:11.405208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.405118 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\" (UniqueName: \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\") pod \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " Oct 30 07:27:11.405208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.405146 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\") pod \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\" (UID: \"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb\") " Oct 30 07:27:11.405761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.405731 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca" pod "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" (UID: "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:11.408019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.407985 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca" pod "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" (UID: "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:11.408133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.407993 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f" pod "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" (UID: "4e0c6c5b-04a2-480f-8ecb-6200b47df8bb"). InnerVolumeSpecName "kube-api-access-kpx5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:11.506380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.506344 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\" (UniqueName: \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\") pod \"a4a2911a-62aa-4753-a002-156ef9013c14\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " Oct 30 07:27:11.506559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.506517 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\") pod \"a4a2911a-62aa-4753-a002-156ef9013c14\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " Oct 30 07:27:11.506686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.506668 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\") pod \"a4a2911a-62aa-4753-a002-156ef9013c14\" (UID: \"a4a2911a-62aa-4753-a002-156ef9013c14\") " Oct 30 07:27:11.506897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.506874 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca" pod "a4a2911a-62aa-4753-a002-156ef9013c14" (UID: "a4a2911a-62aa-4753-a002-156ef9013c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:11.507160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.507136 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpx5f\" (UniqueName: \"kubernetes.io/projected/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-kube-api-access-kpx5f\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.507160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.507155 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.507160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.507164 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.507409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.507173 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.508728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.508705 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5" pod "a4a2911a-62aa-4753-a002-156ef9013c14" (UID: "a4a2911a-62aa-4753-a002-156ef9013c14"). InnerVolumeSpecName "kube-api-access-vpnq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:11.508818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.508754 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca" pod "a4a2911a-62aa-4753-a002-156ef9013c14" (UID: "a4a2911a-62aa-4753-a002-156ef9013c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:11.608246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.608211 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpnq5\" (UniqueName: \"kubernetes.io/projected/a4a2911a-62aa-4753-a002-156ef9013c14-kube-api-access-vpnq5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.608246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.608244 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4a2911a-62aa-4753-a002-156ef9013c14-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:11.804180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:11.804145 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:12.173118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.173038 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" Oct 30 07:27:12.173118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.173071 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x" event={"ID":"a4a2911a-62aa-4753-a002-156ef9013c14","Type":"ContainerDied","Data":"1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e"} Oct 30 07:27:12.173358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.173125 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e4f8ba9965460ec22b43554d4dbe7ff80d1820f1e15d9e584ee3195d07ae24e" Oct 30 07:27:12.174562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.174535 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" event={"ID":"4e0c6c5b-04a2-480f-8ecb-6200b47df8bb","Type":"ContainerDied","Data":"e40796249f8134c7ae03c31df04c23647e77e1959859e0d4d398dc52599afaf1"} Oct 30 07:27:12.174744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.174588 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40796249f8134c7ae03c31df04c23647e77e1959859e0d4d398dc52599afaf1" Oct 30 07:27:12.174744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:12.174542 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh" Oct 30 07:27:13.407470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:13.407436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:15.113874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:15.113839 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:16.791990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:16.791947 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:17.739868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:17.739822 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:27:18.582586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:18.582537 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:20.340100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:20.339744 5894 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 07:27:20.364819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:20.364775 5894 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:20.710602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:20.710505 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:20.819425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:20.819393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.083341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.083306 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4"] Oct 30 07:27:22.084143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084121 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a4a2911a-62aa-4753-a002-156ef9013c14" containerName="revoke-keys" Oct 30 07:27:22.084204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084146 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4a2911a-62aa-4753-a002-156ef9013c14" containerName="revoke-keys" Oct 30 07:27:22.084204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084165 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" containerName="revoke-keys" Oct 30 07:27:22.084204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084174 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" containerName="revoke-keys" Oct 30 07:27:22.084332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084319 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4a2911a-62aa-4753-a002-156ef9013c14" containerName="revoke-keys" Oct 30 07:27:22.084377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.084336 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" containerName="revoke-keys" Oct 30 07:27:22.090087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.090066 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.105137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.105115 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:22.107499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.107472 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4"] Oct 30 07:27:22.115840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.115813 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:22.115978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.115949 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:22.117786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.117762 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:22.209882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.209848 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.210065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.209928 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445th\" (UniqueName: \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.210246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.210200 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.275115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.275078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.311656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.311612 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.311873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.311691 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445th\" (UniqueName: \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.311873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.311811 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.312512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.312480 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.314266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.314238 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.335875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.335437 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-445th\" (UniqueName: \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\") pod \"maas-api-revoke-sub-3debd49f-72cz4\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.404292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.404257 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:22.434034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.433995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.553473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.553339 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.749179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.749153 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4"] Oct 30 07:27:22.849119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.849078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.995268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.995236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:22.995268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.995267 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk"] Oct 30 07:27:22.998066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:22.998041 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-597d57-cxtqk"] Oct 30 07:27:23.111281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:23.111221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:23.288549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:23.288510 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" event={"ID":"40f8dded-a31f-4e52-82d4-b0b636ecb8f3","Type":"ContainerStarted","Data":"cd0f0d2698359e95ddac826e9f72b729b5e45dd29e1c2306a942f499887a0c9d"} Oct 30 07:27:23.288752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:23.288567 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" event={"ID":"40f8dded-a31f-4e52-82d4-b0b636ecb8f3","Type":"ContainerStarted","Data":"2cddcbe8a226e845cb750ccb86f1daff8384433e39ab34111237fa4a041e80b6"} Oct 30 07:27:23.343013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:23.342957 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" podStartSLOduration=1.34294325 podStartE2EDuration="1.34294325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:23.340239057 +0000 UTC m=+1807.031156493" watchObservedRunningTime="2026-10-30 07:27:23.34294325 +0000 UTC m=+1807.033860687" Oct 30 07:27:23.504310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:23.504236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:24.297952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:24.297919 5894 generic.go:356] "Generic (PLEG): container finished" podID="40f8dded-a31f-4e52-82d4-b0b636ecb8f3" containerID="cd0f0d2698359e95ddac826e9f72b729b5e45dd29e1c2306a942f499887a0c9d" exitCode=0 Oct 30 07:27:24.298350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:24.298001 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" event={"ID":"40f8dded-a31f-4e52-82d4-b0b636ecb8f3","Type":"ContainerDied","Data":"cd0f0d2698359e95ddac826e9f72b729b5e45dd29e1c2306a942f499887a0c9d"} Oct 30 07:27:24.432406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:24.432375 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="173c5454-3ebf-40e9-9917-454b0f5e5a46" path="/var/lib/kubelet/pods/173c5454-3ebf-40e9-9917-454b0f5e5a46/volumes" Oct 30 07:27:24.817926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:24.817885 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:25.450335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.450313 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:25.459331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.459304 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\") pod \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " Oct 30 07:27:25.459450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.459346 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\" (UniqueName: \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\") pod \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " Oct 30 07:27:25.459645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.459612 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca" pod "40f8dded-a31f-4e52-82d4-b0b636ecb8f3" (UID: "40f8dded-a31f-4e52-82d4-b0b636ecb8f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:25.459693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.459668 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\") pod \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\" (UID: \"40f8dded-a31f-4e52-82d4-b0b636ecb8f3\") " Oct 30 07:27:25.460286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.460261 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:25.461649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.461614 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca" pod "40f8dded-a31f-4e52-82d4-b0b636ecb8f3" (UID: "40f8dded-a31f-4e52-82d4-b0b636ecb8f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:25.461807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.461788 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th" pod "40f8dded-a31f-4e52-82d4-b0b636ecb8f3" (UID: "40f8dded-a31f-4e52-82d4-b0b636ecb8f3"). InnerVolumeSpecName "kube-api-access-445th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:25.561329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.561250 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-445th\" (UniqueName: \"kubernetes.io/projected/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-kube-api-access-445th\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:25.561329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:25.561278 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40f8dded-a31f-4e52-82d4-b0b636ecb8f3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:26.313163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:26.313135 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" Oct 30 07:27:26.313366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:26.313164 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4" event={"ID":"40f8dded-a31f-4e52-82d4-b0b636ecb8f3","Type":"ContainerDied","Data":"2cddcbe8a226e845cb750ccb86f1daff8384433e39ab34111237fa4a041e80b6"} Oct 30 07:27:26.313366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:26.313246 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cddcbe8a226e845cb750ccb86f1daff8384433e39ab34111237fa4a041e80b6" Oct 30 07:27:26.550349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:26.550316 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:27.285731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.285700 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc"] Oct 30 07:27:27.286568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.286541 5894 cpu_manager.go:414] "containerMap: removing container" podUID="40f8dded-a31f-4e52-82d4-b0b636ecb8f3" containerName="revoke-keys" Oct 30 07:27:27.286568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.286564 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f8dded-a31f-4e52-82d4-b0b636ecb8f3" containerName="revoke-keys" Oct 30 07:27:27.286794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.286675 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f8dded-a31f-4e52-82d4-b0b636ecb8f3" containerName="revoke-keys" Oct 30 07:27:27.290393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.290374 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.306375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.306347 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:27.309939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.309913 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc"] Oct 30 07:27:27.318746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.318705 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:27.318937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.318810 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:27.318937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.318842 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:27.402107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.402067 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.402107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.402109 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxhc\" (UniqueName: \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.402330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.402142 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.503095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.503057 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.503352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.503110 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxhc\" (UniqueName: \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.503352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.503147 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.503921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.503898 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.505627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.505599 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.526760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.526729 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxhc\" (UniqueName: \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\") pod \"maas-api-revoke-sub-c4a1d269-njkmc\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.604407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.604304 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:27.920834 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:27.920796 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676c5fd6_e168_4948_ab8f_8d21b9c69b25.slice/crio-03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8 WatchSource:0}: Error finding container 03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8: Status 404 returned error can't find the container with id 03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8 Oct 30 07:27:27.920992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:27.920913 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc"] Oct 30 07:27:28.035968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.035933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:28.161794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.161754 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:28.278675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.277865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:28.387359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.386132 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" event={"ID":"676c5fd6-e168-4948-ab8f-8d21b9c69b25","Type":"ContainerStarted","Data":"03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8"} Oct 30 07:27:28.640082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.640044 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:28.719712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.719676 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh"] Oct 30 07:27:28.731610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.731581 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-59f7e4-tt9xh"] Oct 30 07:27:28.774016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:28.773955 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:29.266746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:29.266623 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:29.403384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:29.403328 5894 generic.go:356] "Generic (PLEG): container finished" podID="676c5fd6-e168-4948-ab8f-8d21b9c69b25" containerID="7eeda46e583e62d12345693a8bbdd773900d1f613ff9f2853d027432cfeea4f5" exitCode=0 Oct 30 07:27:29.403384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:29.403364 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" event={"ID":"676c5fd6-e168-4948-ab8f-8d21b9c69b25","Type":"ContainerDied","Data":"7eeda46e583e62d12345693a8bbdd773900d1f613ff9f2853d027432cfeea4f5"} Oct 30 07:27:30.136597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.136560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:30.350004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.349969 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:30.432228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.432198 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e0c6c5b-04a2-480f-8ecb-6200b47df8bb" path="/var/lib/kubelet/pods/4e0c6c5b-04a2-480f-8ecb-6200b47df8bb/volumes" Oct 30 07:27:30.556563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.556036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:30.606889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.606864 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:30.647305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.647259 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\") pod \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " Oct 30 07:27:30.647547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.647321 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\" (UniqueName: \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\") pod \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " Oct 30 07:27:30.647987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.647961 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\") pod \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\" (UID: \"676c5fd6-e168-4948-ab8f-8d21b9c69b25\") " Oct 30 07:27:30.648529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.648496 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca" pod "676c5fd6-e168-4948-ab8f-8d21b9c69b25" (UID: "676c5fd6-e168-4948-ab8f-8d21b9c69b25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:30.648805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.648782 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:30.649934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.649891 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc" pod "676c5fd6-e168-4948-ab8f-8d21b9c69b25" (UID: "676c5fd6-e168-4948-ab8f-8d21b9c69b25"). InnerVolumeSpecName "kube-api-access-wdxhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:30.650064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.649971 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca" pod "676c5fd6-e168-4948-ab8f-8d21b9c69b25" (UID: "676c5fd6-e168-4948-ab8f-8d21b9c69b25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:30.681368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.681106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:30.788686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.778415 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/676c5fd6-e168-4948-ab8f-8d21b9c69b25-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:30.788686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:30.778454 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxhc\" (UniqueName: \"kubernetes.io/projected/676c5fd6-e168-4948-ab8f-8d21b9c69b25-kube-api-access-wdxhc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:31.418462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:31.418434 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" Oct 30 07:27:31.418990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:31.418472 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc" event={"ID":"676c5fd6-e168-4948-ab8f-8d21b9c69b25","Type":"ContainerDied","Data":"03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8"} Oct 30 07:27:31.418990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:31.418520 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03187a5b5b67e4ac6fe8a5d2e316bdb68c724f3a7e8d15288297fe78a8be32c8" Oct 30 07:27:31.909188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:31.909149 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:32.021581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:32.021547 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:32.236825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:32.236735 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:32.349622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:32.349592 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:32.805068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:32.805037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:32.935956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:32.935923 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:34.466659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:34.463703 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:35.059065 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:27:35.058858 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:27:35.068872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:35.068834 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:27:35.799336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:35.799304 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:35.913159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:35.913121 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:36.204098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.204003 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:36.320881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.320849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:36.771936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.771893 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj"] Oct 30 07:27:36.772473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.772461 5894 cpu_manager.go:414] "containerMap: removing container" podUID="676c5fd6-e168-4948-ab8f-8d21b9c69b25" containerName="revoke-keys" Oct 30 07:27:36.772533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.772475 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676c5fd6-e168-4948-ab8f-8d21b9c69b25" containerName="revoke-keys" Oct 30 07:27:36.772582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.772573 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="676c5fd6-e168-4948-ab8f-8d21b9c69b25" containerName="revoke-keys" Oct 30 07:27:36.777323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.777302 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.804698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.804666 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:36.805200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.804698 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:36.805308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.804728 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:36.805308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.804778 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:36.808302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.808279 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj"] Oct 30 07:27:36.808682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.808660 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.808937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.808919 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.809007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.808992 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr4cg\" (UniqueName: \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.910368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.910331 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.910580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.910501 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.910580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.910533 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr4cg\" (UniqueName: \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.911125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.911098 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.912916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.912895 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:36.933937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:36.933908 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr4cg\" (UniqueName: \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:37.091195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:37.091160 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:37.400512 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:37.400479 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0458eca3_845b_455b_9b48_987d6514aa13.slice/crio-21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d WatchSource:0}: Error finding container 21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d: Status 404 returned error can't find the container with id 21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d Oct 30 07:27:37.401582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:37.401557 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj"] Oct 30 07:27:37.504433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:37.504396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" event={"ID":"0458eca3-845b-455b-9b48-987d6514aa13","Type":"ContainerStarted","Data":"21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d"} Oct 30 07:27:38.159746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:38.159706 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:38.513390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:38.513304 5894 generic.go:356] "Generic (PLEG): container finished" podID="0458eca3-845b-455b-9b48-987d6514aa13" containerID="4a65d6e2cf507f9cbe50ce637f8f13844fd989dc8f9724670629bc6973c53053" exitCode=0 Oct 30 07:27:38.513390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:38.513362 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" event={"ID":"0458eca3-845b-455b-9b48-987d6514aa13","Type":"ContainerDied","Data":"4a65d6e2cf507f9cbe50ce637f8f13844fd989dc8f9724670629bc6973c53053"} Oct 30 07:27:38.785360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:38.785323 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:39.097198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.097116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:39.680850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.680822 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:39.747116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.747073 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\") pod \"0458eca3-845b-455b-9b48-987d6514aa13\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " Oct 30 07:27:39.747281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.747147 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\") pod \"0458eca3-845b-455b-9b48-987d6514aa13\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " Oct 30 07:27:39.747350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.747290 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\" (UniqueName: \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\") pod \"0458eca3-845b-455b-9b48-987d6514aa13\" (UID: \"0458eca3-845b-455b-9b48-987d6514aa13\") " Oct 30 07:27:39.747542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.747511 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca" pod "0458eca3-845b-455b-9b48-987d6514aa13" (UID: "0458eca3-845b-455b-9b48-987d6514aa13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:39.748150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.748126 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:39.749529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.749501 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg" pod "0458eca3-845b-455b-9b48-987d6514aa13" (UID: "0458eca3-845b-455b-9b48-987d6514aa13"). InnerVolumeSpecName "kube-api-access-wr4cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:39.749733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.749659 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca" pod "0458eca3-845b-455b-9b48-987d6514aa13" (UID: "0458eca3-845b-455b-9b48-987d6514aa13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:39.849717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.849678 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr4cg\" (UniqueName: \"kubernetes.io/projected/0458eca3-845b-455b-9b48-987d6514aa13-kube-api-access-wr4cg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:39.849717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:39.849714 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0458eca3-845b-455b-9b48-987d6514aa13-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:40.145896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.145801 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:40.525614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.525568 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:40.528409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.528387 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" Oct 30 07:27:40.528532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.528382 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj" event={"ID":"0458eca3-845b-455b-9b48-987d6514aa13","Type":"ContainerDied","Data":"21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d"} Oct 30 07:27:40.528532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.528509 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21ed1461d3be9b94f9f9ba8c0e4a0b5e120d23956379a8301b10818979ba4a5d" Oct 30 07:27:40.657345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:40.657308 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:42.050483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:42.050419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:43.832507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:43.832476 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:44.765127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.765083 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc"] Oct 30 07:27:44.765736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.765716 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0458eca3-845b-455b-9b48-987d6514aa13" containerName="revoke-keys" Oct 30 07:27:44.765736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.765736 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0458eca3-845b-455b-9b48-987d6514aa13" containerName="revoke-keys" Oct 30 07:27:44.765958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.765944 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0458eca3-845b-455b-9b48-987d6514aa13" containerName="revoke-keys" Oct 30 07:27:44.769945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.769918 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.786676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.785668 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:44.798041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.798012 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:44.798338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.798322 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:44.798513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.798497 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:44.801700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.801666 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc"] Oct 30 07:27:44.826747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.826714 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.826966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.826944 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txcn4\" (UniqueName: \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.827018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.827000 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.928509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.928470 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.929103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.928592 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.929103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.928802 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txcn4\" (UniqueName: \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.929762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.929736 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.931253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.931230 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:44.952088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:44.952057 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txcn4\" (UniqueName: \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\") pod \"maas-api-revoke-sub-6c947a35-z7gnc\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:45.084048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:45.084015 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:45.394107 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:45.394070 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bd209d_3179_49a9_bf78_204d737783cd.slice/crio-0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c WatchSource:0}: Error finding container 0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c: Status 404 returned error can't find the container with id 0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c Oct 30 07:27:45.394240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:45.394208 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc"] Oct 30 07:27:45.519932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:45.519902 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:45.571448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:45.571412 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" event={"ID":"43bd209d-3179-49a9-bf78-204d737783cd","Type":"ContainerStarted","Data":"0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c"} Oct 30 07:27:46.580326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:46.580292 5894 generic.go:356] "Generic (PLEG): container finished" podID="43bd209d-3179-49a9-bf78-204d737783cd" containerID="83b803bb089adc94ef91ab0646bda54d869b73ad92e7f1a8f45fa141e4839276" exitCode=0 Oct 30 07:27:46.580764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:46.580370 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" event={"ID":"43bd209d-3179-49a9-bf78-204d737783cd","Type":"ContainerDied","Data":"83b803bb089adc94ef91ab0646bda54d869b73ad92e7f1a8f45fa141e4839276"} Oct 30 07:27:47.095156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.095124 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:27:47.099191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.099168 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.123800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.123762 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:27:47.234573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.234539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:47.260754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.260696 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.260950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.260769 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.260950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.260840 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.260960 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.261131 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.261215 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbfx\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.261253 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.261343 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.261429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.261415 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362339 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362424 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362463 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362511 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362556 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362710 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362770 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbfx\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362799 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362844 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.362919 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.363176 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.363427 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.364134 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.364821 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.366856 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.367899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.367431 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.387230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.387199 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbfx\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.390486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.390459 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\") pod \"e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.414161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.414126 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:47.438026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.437989 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:47.566980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.566882 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:47.720945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.720913 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:47.750954 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:27:47.750924 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea7dd5e1_0470_46d2_8f35_4ddac39b881f.slice/crio-983c5a177106cb0539873ba683b22bfcaabd4bf41fa9b786a111c333c7a934c5 WatchSource:0}: Error finding container 983c5a177106cb0539873ba683b22bfcaabd4bf41fa9b786a111c333c7a934c5: Status 404 returned error can't find the container with id 983c5a177106cb0539873ba683b22bfcaabd4bf41fa9b786a111c333c7a934c5 Oct 30 07:27:47.752372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.752348 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:27:47.753950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.753920 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:47.754016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.753986 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:47.754053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.754014 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:27:47.868358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.868318 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\") pod \"43bd209d-3179-49a9-bf78-204d737783cd\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " Oct 30 07:27:47.868540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.868509 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\") pod \"43bd209d-3179-49a9-bf78-204d737783cd\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " Oct 30 07:27:47.868592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.868572 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\" (UniqueName: \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\") pod \"43bd209d-3179-49a9-bf78-204d737783cd\" (UID: \"43bd209d-3179-49a9-bf78-204d737783cd\") " Oct 30 07:27:47.868739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.868700 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca" pod "43bd209d-3179-49a9-bf78-204d737783cd" (UID: "43bd209d-3179-49a9-bf78-204d737783cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:27:47.869097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.869070 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:47.870580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.870556 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca" pod "43bd209d-3179-49a9-bf78-204d737783cd" (UID: "43bd209d-3179-49a9-bf78-204d737783cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:27:47.870834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.870813 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4" pod "43bd209d-3179-49a9-bf78-204d737783cd" (UID: "43bd209d-3179-49a9-bf78-204d737783cd"). InnerVolumeSpecName "kube-api-access-txcn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:27:47.978772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.978675 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43bd209d-3179-49a9-bf78-204d737783cd-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:47.978772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:47.978716 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txcn4\" (UniqueName: \"kubernetes.io/projected/43bd209d-3179-49a9-bf78-204d737783cd-kube-api-access-txcn4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:27:48.597896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.597869 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" Oct 30 07:27:48.598082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.597863 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc" event={"ID":"43bd209d-3179-49a9-bf78-204d737783cd","Type":"ContainerDied","Data":"0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c"} Oct 30 07:27:48.598082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.597972 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b71aab677d7d3608c0ca379e31388c53391c8346e9e086e1a1d135b7e69b65c" Oct 30 07:27:48.599912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.599884 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" event={"ID":"ea7dd5e1-0470-46d2-8f35-4ddac39b881f","Type":"ContainerStarted","Data":"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1"} Oct 30 07:27:48.600057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.599935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" event={"ID":"ea7dd5e1-0470-46d2-8f35-4ddac39b881f","Type":"ContainerStarted","Data":"983c5a177106cb0539873ba683b22bfcaabd4bf41fa9b786a111c333c7a934c5"} Oct 30 07:27:48.658760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:48.658624 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podStartSLOduration=1.658605015 podStartE2EDuration="1.658605015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:27:48.658245085 +0000 UTC m=+1832.349162524" watchObservedRunningTime="2026-10-30 07:27:48.658605015 +0000 UTC m=+1832.349522452" Oct 30 07:27:49.016157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:49.016062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:49.414786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:49.414758 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:49.416397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:49.416374 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:49.416529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:49.416419 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:50.415480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:50.415449 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:50.415909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:50.415501 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:50.754664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:50.754548 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:51.415562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:51.415528 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:51.416013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:51.415586 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:52.227951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.227890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:52.416859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.416816 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:52.417475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.417440 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:52.796869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.796753 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:52.937369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.936959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:52.970057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:52.970023 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj"] Oct 30 07:27:53.005267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.005230 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4a665-dslnj"] Oct 30 07:27:53.054181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.054104 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:53.164583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.164553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:53.414897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.414816 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:53.414897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.414871 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:53.561996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:53.561965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:54.415121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:54.415078 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 07:27:54.415312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:54.415153 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 07:27:54.432876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:54.432842 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0458eca3-845b-455b-9b48-987d6514aa13" path="/var/lib/kubelet/pods/0458eca3-845b-455b-9b48-987d6514aa13/volumes" Oct 30 07:27:54.926507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:54.926467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:55.451817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:55.451772 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:27:55.595794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:55.595762 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:55.596471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:55.596443 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:55.597541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:55.597520 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:27:56.998341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:56.998272 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:59.236975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.236939 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:59.591494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.591460 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4"] Oct 30 07:27:59.592056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.592041 5894 cpu_manager.go:414] "containerMap: removing container" podUID="43bd209d-3179-49a9-bf78-204d737783cd" containerName="revoke-keys" Oct 30 07:27:59.592136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.592056 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd209d-3179-49a9-bf78-204d737783cd" containerName="revoke-keys" Oct 30 07:27:59.592202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.592192 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd209d-3179-49a9-bf78-204d737783cd" containerName="revoke-keys" Oct 30 07:27:59.596830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.596807 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.616500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.616467 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4"] Oct 30 07:27:59.624472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.624439 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:27:59.625679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.624692 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:27:59.625679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.624700 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:27:59.625679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.624833 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:27:59.655863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.655814 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6j5k\" (UniqueName: \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.656148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.656120 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.656204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.656166 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.750702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.750662 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:59.758032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.758000 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6j5k\" (UniqueName: \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.758182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.758160 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.758225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.758181 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.758983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.758948 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.760670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.760650 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.781857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.781828 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6j5k\" (UniqueName: \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\") pod \"maas-api-revoke-sub-85e7fb5e-qrbj4\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.900337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.900259 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:27:59.912383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.912355 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:27:59.948975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.948944 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th"] Oct 30 07:27:59.954430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.954404 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:27:59.976949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:27:59.976913 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th"] Oct 30 07:28:00.056327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.056282 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8"] Oct 30 07:28:00.061357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.061331 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.063856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.063765 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.063856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.063828 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcn9b\" (UniqueName: \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.064068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.064019 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.105663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.091605 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8"] Oct 30 07:28:00.135713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.135679 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:00.165911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.165812 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk4q7\" (UniqueName: \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.166099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.165963 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.166099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.166018 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.166099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.166041 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.166099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.166060 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcn9b\" (UniqueName: \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.166343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.166196 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.166887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.166865 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.168312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.168288 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.190095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.190066 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcn9b\" (UniqueName: \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\") pod \"maas-api-revoke-sub-64f12108-k92th\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.270043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.268914 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk4q7\" (UniqueName: \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.270043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.269055 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.270043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.269101 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.270989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.270962 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.277874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.275172 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4"] Oct 30 07:28:00.281746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.280371 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.284579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.284545 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:00.295709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.295440 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk4q7\" (UniqueName: \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\") pod \"maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.396518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.396489 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:00.610746 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:00.610710 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b243156_786c_4491_90ff_53d793280097.slice/crio-412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1 WatchSource:0}: Error finding container 412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1: Status 404 returned error can't find the container with id 412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1 Oct 30 07:28:00.612751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.612728 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th"] Oct 30 07:28:00.711122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.711092 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8"] Oct 30 07:28:00.773665 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:00.773609 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5e8000_8140_4220_adaf_d26df078aca1.slice/crio-624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574 WatchSource:0}: Error finding container 624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574: Status 404 returned error can't find the container with id 624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574 Oct 30 07:28:00.854699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.854665 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" event={"ID":"4e5e8000-8140-4220-adaf-d26df078aca1","Type":"ContainerStarted","Data":"624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574"} Oct 30 07:28:00.856313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.856284 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" event={"ID":"d35f67f3-68c9-4ad9-b8d3-170deb65e034","Type":"ContainerStarted","Data":"2e72a6a28d3b6eba92601b77515ed2edf0c054afc0e5c39197d2c5b2aec11924"} Oct 30 07:28:00.856455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.856333 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" event={"ID":"d35f67f3-68c9-4ad9-b8d3-170deb65e034","Type":"ContainerStarted","Data":"d41a9eca4fd86ffbee0e7b56acc4013d8d656bce8819c3fdfeeb04a4177c3981"} Oct 30 07:28:00.857702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.857676 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" event={"ID":"4b243156-786c-4491-90ff-53d793280097","Type":"ContainerStarted","Data":"412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1"} Oct 30 07:28:00.913392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:00.913335 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" podStartSLOduration=1.9133179120000001 podStartE2EDuration="1.913317912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:00.911780746 +0000 UTC m=+1844.602698184" watchObservedRunningTime="2026-10-30 07:28:00.913317912 +0000 UTC m=+1844.604235411" Oct 30 07:28:01.275487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.275399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:01.870712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.870600 5894 generic.go:356] "Generic (PLEG): container finished" podID="4e5e8000-8140-4220-adaf-d26df078aca1" containerID="75ebecae3265cd0fe9dd084a01119d638ea3f1f5aa217fe89f141159f259404a" exitCode=0 Oct 30 07:28:01.870712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.870668 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" event={"ID":"4e5e8000-8140-4220-adaf-d26df078aca1","Type":"ContainerDied","Data":"75ebecae3265cd0fe9dd084a01119d638ea3f1f5aa217fe89f141159f259404a"} Oct 30 07:28:01.872361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.872334 5894 generic.go:356] "Generic (PLEG): container finished" podID="d35f67f3-68c9-4ad9-b8d3-170deb65e034" containerID="2e72a6a28d3b6eba92601b77515ed2edf0c054afc0e5c39197d2c5b2aec11924" exitCode=0 Oct 30 07:28:01.872499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.872416 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" event={"ID":"d35f67f3-68c9-4ad9-b8d3-170deb65e034","Type":"ContainerDied","Data":"2e72a6a28d3b6eba92601b77515ed2edf0c054afc0e5c39197d2c5b2aec11924"} Oct 30 07:28:01.874158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.874135 5894 generic.go:356] "Generic (PLEG): container finished" podID="4b243156-786c-4491-90ff-53d793280097" containerID="a6ebe8829c92675027680d7f6e888689cad20523408f20e31eb1a8351cd925b3" exitCode=0 Oct 30 07:28:01.874232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:01.874187 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" event={"ID":"4b243156-786c-4491-90ff-53d793280097","Type":"ContainerDied","Data":"a6ebe8829c92675027680d7f6e888689cad20523408f20e31eb1a8351cd925b3"} Oct 30 07:28:03.008487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.008401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:03.129563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.129531 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:28:03.184312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.179866 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:03.184312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.181991 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:03.221074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.221042 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\") pod \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " Oct 30 07:28:03.221291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.221265 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\" (UniqueName: \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\") pod \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " Oct 30 07:28:03.221365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.221337 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\") pod \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\" (UID: \"d35f67f3-68c9-4ad9-b8d3-170deb65e034\") " Oct 30 07:28:03.221422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.221377 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca" pod "d35f67f3-68c9-4ad9-b8d3-170deb65e034" (UID: "d35f67f3-68c9-4ad9-b8d3-170deb65e034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:03.221952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.221926 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.224218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.224185 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca" pod "d35f67f3-68c9-4ad9-b8d3-170deb65e034" (UID: "d35f67f3-68c9-4ad9-b8d3-170deb65e034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:03.228227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.228193 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k" pod "d35f67f3-68c9-4ad9-b8d3-170deb65e034" (UID: "d35f67f3-68c9-4ad9-b8d3-170deb65e034"). InnerVolumeSpecName "kube-api-access-c6j5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:03.322988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.322956 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\") pod \"4b243156-786c-4491-90ff-53d793280097\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " Oct 30 07:28:03.323171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323023 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\") pod \"4b243156-786c-4491-90ff-53d793280097\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " Oct 30 07:28:03.323171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\" (UniqueName: \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\") pod \"4e5e8000-8140-4220-adaf-d26df078aca1\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " Oct 30 07:28:03.323344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323179 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\") pod \"4e5e8000-8140-4220-adaf-d26df078aca1\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " Oct 30 07:28:03.323344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323232 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\") pod \"4e5e8000-8140-4220-adaf-d26df078aca1\" (UID: \"4e5e8000-8140-4220-adaf-d26df078aca1\") " Oct 30 07:28:03.323344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323259 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\" (UniqueName: \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\") pod \"4b243156-786c-4491-90ff-53d793280097\" (UID: \"4b243156-786c-4491-90ff-53d793280097\") " Oct 30 07:28:03.323509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323463 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca" pod "4b243156-786c-4491-90ff-53d793280097" (UID: "4b243156-786c-4491-90ff-53d793280097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:03.323662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323622 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca" pod "4e5e8000-8140-4220-adaf-d26df078aca1" (UID: "4e5e8000-8140-4220-adaf-d26df078aca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:03.323890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323870 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d35f67f3-68c9-4ad9-b8d3-170deb65e034-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.324038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323894 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.324038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323909 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b243156-786c-4491-90ff-53d793280097-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.324038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.323923 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6j5k\" (UniqueName: \"kubernetes.io/projected/d35f67f3-68c9-4ad9-b8d3-170deb65e034-kube-api-access-c6j5k\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.325500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.325473 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7" pod "4e5e8000-8140-4220-adaf-d26df078aca1" (UID: "4e5e8000-8140-4220-adaf-d26df078aca1"). InnerVolumeSpecName "kube-api-access-gk4q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:03.325604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.325522 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca" pod "4b243156-786c-4491-90ff-53d793280097" (UID: "4b243156-786c-4491-90ff-53d793280097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:03.325828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.325803 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca" pod "4e5e8000-8140-4220-adaf-d26df078aca1" (UID: "4e5e8000-8140-4220-adaf-d26df078aca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:03.325883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.325841 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b" pod "4b243156-786c-4491-90ff-53d793280097" (UID: "4b243156-786c-4491-90ff-53d793280097"). InnerVolumeSpecName "kube-api-access-rcn9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:03.424776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.424696 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk4q7\" (UniqueName: \"kubernetes.io/projected/4e5e8000-8140-4220-adaf-d26df078aca1-kube-api-access-gk4q7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.424776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.424728 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e5e8000-8140-4220-adaf-d26df078aca1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.424776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.424742 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcn9b\" (UniqueName: \"kubernetes.io/projected/4b243156-786c-4491-90ff-53d793280097-kube-api-access-rcn9b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.424776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.424758 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b243156-786c-4491-90ff-53d793280097-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:03.902303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.902000 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" Oct 30 07:28:03.902303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.902023 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8" event={"ID":"4e5e8000-8140-4220-adaf-d26df078aca1","Type":"ContainerDied","Data":"624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574"} Oct 30 07:28:03.902303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.902079 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624fb57000e88b84aa0ec79ff63746f4560fbcfc59e375e16a3f288691ab5574" Oct 30 07:28:03.903574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.903472 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" Oct 30 07:28:03.903574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.903512 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4" event={"ID":"d35f67f3-68c9-4ad9-b8d3-170deb65e034","Type":"ContainerDied","Data":"d41a9eca4fd86ffbee0e7b56acc4013d8d656bce8819c3fdfeeb04a4177c3981"} Oct 30 07:28:03.903574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.903554 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d41a9eca4fd86ffbee0e7b56acc4013d8d656bce8819c3fdfeeb04a4177c3981" Oct 30 07:28:03.905020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.905003 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" Oct 30 07:28:03.905115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.905050 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th" event={"ID":"4b243156-786c-4491-90ff-53d793280097","Type":"ContainerDied","Data":"412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1"} Oct 30 07:28:03.905115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:03.905098 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412fe6abf7b9a4f05bf64d85cdb1d50d370a0c6c261d13f318076cffc9ff12c1" Oct 30 07:28:04.816278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:04.816243 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:04.828933 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:28:04.828900 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:28:06.429058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.427656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:06.711865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.711778 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g"] Oct 30 07:28:06.712441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712420 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4e5e8000-8140-4220-adaf-d26df078aca1" containerName="revoke-keys" Oct 30 07:28:06.712550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712441 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e5e8000-8140-4220-adaf-d26df078aca1" containerName="revoke-keys" Oct 30 07:28:06.712550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712503 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4b243156-786c-4491-90ff-53d793280097" containerName="revoke-keys" Oct 30 07:28:06.712550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712511 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b243156-786c-4491-90ff-53d793280097" containerName="revoke-keys" Oct 30 07:28:06.712550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712527 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d35f67f3-68c9-4ad9-b8d3-170deb65e034" containerName="revoke-keys" Oct 30 07:28:06.712550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712534 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35f67f3-68c9-4ad9-b8d3-170deb65e034" containerName="revoke-keys" Oct 30 07:28:06.712837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712692 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b243156-786c-4491-90ff-53d793280097" containerName="revoke-keys" Oct 30 07:28:06.712837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712705 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35f67f3-68c9-4ad9-b8d3-170deb65e034" containerName="revoke-keys" Oct 30 07:28:06.712837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.712718 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e5e8000-8140-4220-adaf-d26df078aca1" containerName="revoke-keys" Oct 30 07:28:06.718156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.718134 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.736048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.736020 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g"] Oct 30 07:28:06.744334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.744303 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:06.744489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.744345 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:06.744489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.744372 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:06.745609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.745588 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:06.868807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.868764 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8lm\" (UniqueName: \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.869005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.868897 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.869005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.868959 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.970252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.970155 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.970252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.970239 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.970500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.970421 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8lm\" (UniqueName: \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.970962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.970941 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.972861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.972839 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:06.994180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:06.994151 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8lm\" (UniqueName: \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\") pod \"maas-api-revoke-sub-8c6bda3c-zc24g\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:07.022670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.022620 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:28:07.022866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.022843 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" containerID="cri-o://cad9f4ad5500e093a9fe19ac80a439f335b2f720349f4f7b4fedf562aa20aece" gracePeriod=30 Oct 30 07:28:07.033503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.033470 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:07.345739 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:07.345704 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef3436c9_8dfc_4cab_8b98_086711d6484b.slice/crio-afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45 WatchSource:0}: Error finding container afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45: Status 404 returned error can't find the container with id afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45 Oct 30 07:28:07.347277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.347249 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g"] Oct 30 07:28:07.937882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.937792 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" event={"ID":"ef3436c9-8dfc-4cab-8b98-086711d6484b","Type":"ContainerStarted","Data":"705e2984306542f7b8fbde03c6164e573aaf400d64b11cee1f012a9381c9b52c"} Oct 30 07:28:07.937882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.937853 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" event={"ID":"ef3436c9-8dfc-4cab-8b98-086711d6484b","Type":"ContainerStarted","Data":"afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45"} Oct 30 07:28:07.984141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:07.984089 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" podStartSLOduration=1.98407384 podStartE2EDuration="1.98407384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:07.980708717 +0000 UTC m=+1851.671626156" watchObservedRunningTime="2026-10-30 07:28:07.98407384 +0000 UTC m=+1851.674991277" Oct 30 07:28:08.130796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:08.130753 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:08.946863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:08.946829 5894 generic.go:356] "Generic (PLEG): container finished" podID="ef3436c9-8dfc-4cab-8b98-086711d6484b" containerID="705e2984306542f7b8fbde03c6164e573aaf400d64b11cee1f012a9381c9b52c" exitCode=0 Oct 30 07:28:08.947251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:08.946881 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" event={"ID":"ef3436c9-8dfc-4cab-8b98-086711d6484b","Type":"ContainerDied","Data":"705e2984306542f7b8fbde03c6164e573aaf400d64b11cee1f012a9381c9b52c"} Oct 30 07:28:09.830357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:09.830321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:10.199700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.199669 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:10.346755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.345447 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\") pod \"ef3436c9-8dfc-4cab-8b98-086711d6484b\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " Oct 30 07:28:10.346755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.345525 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\") pod \"ef3436c9-8dfc-4cab-8b98-086711d6484b\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " Oct 30 07:28:10.346755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.345570 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\" (UniqueName: \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\") pod \"ef3436c9-8dfc-4cab-8b98-086711d6484b\" (UID: \"ef3436c9-8dfc-4cab-8b98-086711d6484b\") " Oct 30 07:28:10.347101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.346979 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca" pod "ef3436c9-8dfc-4cab-8b98-086711d6484b" (UID: "ef3436c9-8dfc-4cab-8b98-086711d6484b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:10.396659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.383882 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca" pod "ef3436c9-8dfc-4cab-8b98-086711d6484b" (UID: "ef3436c9-8dfc-4cab-8b98-086711d6484b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:10.400054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.398852 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm" pod "ef3436c9-8dfc-4cab-8b98-086711d6484b" (UID: "ef3436c9-8dfc-4cab-8b98-086711d6484b"). InnerVolumeSpecName "kube-api-access-vr8lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:10.447264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.447227 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:10.447264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.447266 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef3436c9-8dfc-4cab-8b98-086711d6484b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:10.447496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:10.447280 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8lm\" (UniqueName: \"kubernetes.io/projected/ef3436c9-8dfc-4cab-8b98-086711d6484b-kube-api-access-vr8lm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:11.158666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:11.134160 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:11.300653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:11.300572 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" Oct 30 07:28:11.301128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:11.300711 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g" event={"ID":"ef3436c9-8dfc-4cab-8b98-086711d6484b","Type":"ContainerDied","Data":"afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45"} Oct 30 07:28:11.301128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:11.300955 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afc28fc4e8105ad133291542abbe6b733351fbbe72573cdf38c091dde9c15f45" Oct 30 07:28:11.422717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:11.398308 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:12.039997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.039938 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.051893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.051838 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.090588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.089733 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.221607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.221565 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded" Oct 30 07:28:12.378718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.378325 5894 generic.go:356] "Generic (PLEG): container finished" podID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerID="cad9f4ad5500e093a9fe19ac80a439f335b2f720349f4f7b4fedf562aa20aece" exitCode=0 Oct 30 07:28:12.379430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.379343 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" event={"ID":"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0","Type":"ContainerDied","Data":"cad9f4ad5500e093a9fe19ac80a439f335b2f720349f4f7b4fedf562aa20aece"} Oct 30 07:28:12.505653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.494974 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:12.936520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.936412 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:28:12.975737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.975702 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:28:12.999654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:12.999340 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:28:13.131444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.131403 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.132155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.132129 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.135220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.135196 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.137575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.137550 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.149379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.149350 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.149915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.149894 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.153188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.153157 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.153292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.153222 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.153348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.153339 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\") pod \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\" (UID: \"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0\") " Oct 30 07:28:13.155996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.155964 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:13.156563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.156535 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:13.156895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.156874 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:13.157187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.157167 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:13.159760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.159738 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:13.225306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.223514 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "kube-api-access-lmbg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:13.225306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.223751 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:28:13.225306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.224010 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:28:13.225867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.225839 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token" pod "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" (UID: "da06143b-f46e-4b7e-a6f2-e71ba6a1fad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:13.279650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.279604 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.279919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.279901 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280039 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280136 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280224 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280304 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280396 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280505 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmbg7\" (UniqueName: \"kubernetes.io/projected/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-kube-api-access-lmbg7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.280603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.280592 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:13.404557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.401861 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" event={"ID":"da06143b-f46e-4b7e-a6f2-e71ba6a1fad0","Type":"ContainerDied","Data":"ad11a20e18ade2fd23bdb8572f155f1d47886f4dd83a3c8f76529917e7b9d135"} Oct 30 07:28:13.404557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.404424 5894 scope.go:122] "RemoveContainer" containerID="cad9f4ad5500e093a9fe19ac80a439f335b2f720349f4f7b4fedf562aa20aece" Oct 30 07:28:13.406485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.406459 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs" Oct 30 07:28:13.463446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.463370 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:28:13.470311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:13.470283 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-pdmqs"] Oct 30 07:28:14.251996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.251964 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:14.431878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.431847 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" path="/var/lib/kubelet/pods/da06143b-f46e-4b7e-a6f2-e71ba6a1fad0/volumes" Oct 30 07:28:14.621187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.621147 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5"] Oct 30 07:28:14.621863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.621838 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ef3436c9-8dfc-4cab-8b98-086711d6484b" containerName="revoke-keys" Oct 30 07:28:14.621863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.621862 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef3436c9-8dfc-4cab-8b98-086711d6484b" containerName="revoke-keys" Oct 30 07:28:14.622081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.621957 5894 cpu_manager.go:414] "containerMap: removing container" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" Oct 30 07:28:14.622081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.621967 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" Oct 30 07:28:14.622186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.622087 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="da06143b-f46e-4b7e-a6f2-e71ba6a1fad0" containerName="istio-proxy" Oct 30 07:28:14.622186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.622104 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef3436c9-8dfc-4cab-8b98-086711d6484b" containerName="revoke-keys" Oct 30 07:28:14.627647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.627606 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.645211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.645183 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5"] Oct 30 07:28:14.653318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.653289 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:14.653523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.653339 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:14.657433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.657413 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:14.657561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.657453 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:14.699909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.699872 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.700092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.699994 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.700092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.700058 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8x6\" (UniqueName: \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.801130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.801092 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8x6\" (UniqueName: \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.801296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.801281 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.801378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.801354 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.802136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.802113 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.804016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.803982 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.822258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.822230 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z"] Oct 30 07:28:14.826482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.826464 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:14.834480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.834457 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8x6\" (UniqueName: \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\") pod \"maas-api-revoke-sub-f51dcef3-7r5j5\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:14.846729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.846699 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z"] Oct 30 07:28:14.902750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.902662 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:14.902750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.902711 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:14.903101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.903070 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfl2\" (UniqueName: \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:14.953505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:14.953471 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:15.004805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.004762 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfl2\" (UniqueName: \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.004981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.004855 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.004981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.004894 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.005847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.005799 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.007373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.007345 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.046099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.046063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfl2\" (UniqueName: \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\") pod \"maas-api-revoke-sub-d4309d09-fh75z\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.123935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.123891 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:15.163505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.163421 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:15.269747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.269706 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5"] Oct 30 07:28:15.271471 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:15.271440 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3640ee90_6dd7_4bda_af5e_eb4eb6e0a66f.slice/crio-ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2 WatchSource:0}: Error finding container ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2: Status 404 returned error can't find the container with id ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2 Oct 30 07:28:15.426164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.426074 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" event={"ID":"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f","Type":"ContainerStarted","Data":"ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2"} Oct 30 07:28:15.457802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.457760 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:15.475487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.475454 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z"] Oct 30 07:28:15.476517 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:15.476492 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fc17ef_e735_40f1_8325_064ddacb3719.slice/crio-e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d WatchSource:0}: Error finding container e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d: Status 404 returned error can't find the container with id e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d Oct 30 07:28:15.937091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:15.936993 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:16.059054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.059017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:16.436301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.436267 5894 generic.go:356] "Generic (PLEG): container finished" podID="3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" containerID="a493204ab1e6e8d0fd8cac9af815d7a05b2d8f4ae90fd6b9b2fd4674e88bc39d" exitCode=0 Oct 30 07:28:16.436459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.436319 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" event={"ID":"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f","Type":"ContainerDied","Data":"a493204ab1e6e8d0fd8cac9af815d7a05b2d8f4ae90fd6b9b2fd4674e88bc39d"} Oct 30 07:28:16.438116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.438084 5894 generic.go:356] "Generic (PLEG): container finished" podID="10fc17ef-e735-40f1-8325-064ddacb3719" containerID="c72de33844d28f795133c1a3040cfee37e05da9e7100c7525f667822788f6456" exitCode=0 Oct 30 07:28:16.438116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.438114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" event={"ID":"10fc17ef-e735-40f1-8325-064ddacb3719","Type":"ContainerDied","Data":"c72de33844d28f795133c1a3040cfee37e05da9e7100c7525f667822788f6456"} Oct 30 07:28:16.438261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.438146 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" event={"ID":"10fc17ef-e735-40f1-8325-064ddacb3719","Type":"ContainerStarted","Data":"e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d"} Oct 30 07:28:16.809563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:16.809524 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:17.651348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.651315 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:17.654445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.654421 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:17.752137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.752042 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\") pod \"10fc17ef-e735-40f1-8325-064ddacb3719\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " Oct 30 07:28:17.752137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.752118 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\" (UniqueName: \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\") pod \"10fc17ef-e735-40f1-8325-064ddacb3719\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " Oct 30 07:28:17.752814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.752793 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\") pod \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " Oct 30 07:28:17.752917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.752848 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\" (UniqueName: \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\") pod \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " Oct 30 07:28:17.752974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.752957 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\") pod \"10fc17ef-e735-40f1-8325-064ddacb3719\" (UID: \"10fc17ef-e735-40f1-8325-064ddacb3719\") " Oct 30 07:28:17.753082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.753062 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\") pod \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\" (UID: \"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f\") " Oct 30 07:28:17.753589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.753352 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca" pod "10fc17ef-e735-40f1-8325-064ddacb3719" (UID: "10fc17ef-e735-40f1-8325-064ddacb3719"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:17.753589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.753568 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca" pod "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" (UID: "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:17.753857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.753833 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.753921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.753862 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.755676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.755606 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2" pod "10fc17ef-e735-40f1-8325-064ddacb3719" (UID: "10fc17ef-e735-40f1-8325-064ddacb3719"). InnerVolumeSpecName "kube-api-access-qdfl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:17.755853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.755836 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca" pod "10fc17ef-e735-40f1-8325-064ddacb3719" (UID: "10fc17ef-e735-40f1-8325-064ddacb3719"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:17.756138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.756114 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6" pod "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" (UID: "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f"). InnerVolumeSpecName "kube-api-access-2b8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:17.757118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.757093 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca" pod "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" (UID: "3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:17.854854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.854820 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fc17ef-e735-40f1-8325-064ddacb3719-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.854854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.854850 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfl2\" (UniqueName: \"kubernetes.io/projected/10fc17ef-e735-40f1-8325-064ddacb3719-kube-api-access-qdfl2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.854854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.854861 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8x6\" (UniqueName: \"kubernetes.io/projected/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-kube-api-access-2b8x6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.855325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.854873 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:17.879920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.879893 5894 scope.go:122] "RemoveContainer" containerID="e7d34020f7988d8357f90bb583baf65caa7126369656c7d5c747fa73c946485a" Oct 30 07:28:17.890617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:17.890587 5894 scope.go:122] "RemoveContainer" containerID="0bf0e7ded0b68bee1cf572b119979993059b92523d602bb516c43e5c3042e87c" Oct 30 07:28:18.456073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.456038 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" Oct 30 07:28:18.456073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.456053 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z" event={"ID":"10fc17ef-e735-40f1-8325-064ddacb3719","Type":"ContainerDied","Data":"e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d"} Oct 30 07:28:18.456278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.456105 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e95ed6f74690fab43ebdae24baec17b0687decd3d38c1ea829197f755cc43c5d" Oct 30 07:28:18.457616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.457594 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" Oct 30 07:28:18.457732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.457592 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5" event={"ID":"3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f","Type":"ContainerDied","Data":"ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2"} Oct 30 07:28:18.457732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.457696 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef7396cd8709661500ded49e6e75ff1a2c252d151c36ebf4e6dca9898e5fdec2" Oct 30 07:28:18.645996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:18.645961 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:19.219289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219251 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8"] Oct 30 07:28:19.219803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219787 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" containerName="revoke-keys" Oct 30 07:28:19.219803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219802 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" containerName="revoke-keys" Oct 30 07:28:19.219934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219822 5894 cpu_manager.go:414] "containerMap: removing container" podUID="10fc17ef-e735-40f1-8325-064ddacb3719" containerName="revoke-keys" Oct 30 07:28:19.219934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219827 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fc17ef-e735-40f1-8325-064ddacb3719" containerName="revoke-keys" Oct 30 07:28:19.219934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219933 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" containerName="revoke-keys" Oct 30 07:28:19.220081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.219943 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fc17ef-e735-40f1-8325-064ddacb3719" containerName="revoke-keys" Oct 30 07:28:19.225024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.225004 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.242918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.242884 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8"] Oct 30 07:28:19.251963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.251883 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:19.251963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.251883 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:19.279300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.279270 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:19.279460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.279309 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:19.375917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.375887 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.376111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.376091 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.376209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.376132 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554g5\" (UniqueName: \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.477426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.477342 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.477426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.477381 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554g5\" (UniqueName: \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.477682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.477490 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.478068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.478045 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.479913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.479884 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.501366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.501336 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554g5\" (UniqueName: \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\") pod \"maas-api-revoke-sub-925c0e98-p4ln8\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.541102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.541068 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:19.846350 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:19.846305 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80457bf1_3bfd_45ee_828a_fe6dd1d7a8d3.slice/crio-0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4 WatchSource:0}: Error finding container 0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4: Status 404 returned error can't find the container with id 0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4 Oct 30 07:28:19.847322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:19.847297 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8"] Oct 30 07:28:20.279211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.279172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:20.401329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.401293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:20.475507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.475452 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" event={"ID":"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3","Type":"ContainerStarted","Data":"8cdf61859eba48c1b06e2d8645d63e5679574170226edf5a0c8621be5bdb34ef"} Oct 30 07:28:20.475507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.475517 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" event={"ID":"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3","Type":"ContainerStarted","Data":"0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4"} Oct 30 07:28:20.523511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.523453 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" podStartSLOduration=1.523437612 podStartE2EDuration="1.523437612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:20.521424353 +0000 UTC m=+1864.212341791" watchObservedRunningTime="2026-10-30 07:28:20.523437612 +0000 UTC m=+1864.214355049" Oct 30 07:28:20.637404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:20.637319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:21.487823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.487785 5894 generic.go:356] "Generic (PLEG): container finished" podID="80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" containerID="8cdf61859eba48c1b06e2d8645d63e5679574170226edf5a0c8621be5bdb34ef" exitCode=0 Oct 30 07:28:21.488208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.487867 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" event={"ID":"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3","Type":"ContainerDied","Data":"8cdf61859eba48c1b06e2d8645d63e5679574170226edf5a0c8621be5bdb34ef"} Oct 30 07:28:21.543648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.543602 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6"] Oct 30 07:28:21.547870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.547844 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.568714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.564858 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6"] Oct 30 07:28:21.706779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.706735 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.706975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.706845 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.707037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.706987 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-894xl\" (UniqueName: \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.808691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.808616 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.808874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.808739 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.808874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.808867 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-894xl\" (UniqueName: \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.809429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.809409 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.811047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.811028 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.864284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.864246 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-894xl\" (UniqueName: \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\") pod \"maas-api-revoke-sub-98fec99a-qclm6\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:21.877226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:21.877192 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:22.180575 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:22.180540 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636c5d5e_6e95_45ed_8fd0_d303fc792a8a.slice/crio-4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116 WatchSource:0}: Error finding container 4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116: Status 404 returned error can't find the container with id 4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116 Oct 30 07:28:22.181530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.181497 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6"] Oct 30 07:28:22.313712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.313661 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:22.437648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.437534 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:22.497512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.497476 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" event={"ID":"636c5d5e-6e95-45ed-8fd0-d303fc792a8a","Type":"ContainerStarted","Data":"d79b4798091f154d5512844f87afdc743c639adb7c00a41661e1bebfdcbbf7c3"} Oct 30 07:28:22.497947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.497534 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" event={"ID":"636c5d5e-6e95-45ed-8fd0-d303fc792a8a","Type":"ContainerStarted","Data":"4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116"} Oct 30 07:28:22.546269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.546180 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" podStartSLOduration=1.546165331 podStartE2EDuration="1.546165331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:22.543794436 +0000 UTC m=+1866.234711872" watchObservedRunningTime="2026-10-30 07:28:22.546165331 +0000 UTC m=+1866.237082768" Oct 30 07:28:22.677682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.677436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:22.731372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.731344 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:22.773588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.773555 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\") pod \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " Oct 30 07:28:22.774110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.773911 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\") pod \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " Oct 30 07:28:22.774110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.773982 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\" (UniqueName: \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\") pod \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\" (UID: \"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3\") " Oct 30 07:28:22.775697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.775615 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca" pod "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" (UID: "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:22.778059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.778020 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5" pod "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" (UID: "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3"). InnerVolumeSpecName "kube-api-access-554g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:22.779659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.779130 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca" pod "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" (UID: "80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:22.875596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.875560 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.875596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.875593 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554g5\" (UniqueName: \"kubernetes.io/projected/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-kube-api-access-554g5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:22.875596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:22.875603 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:23.506208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:23.506117 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" Oct 30 07:28:23.506208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:23.506138 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8" event={"ID":"80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3","Type":"ContainerDied","Data":"0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4"} Oct 30 07:28:23.506208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:23.506186 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d72613253c24c03dc228e9e301050c4bffd59b6d4927bbcdf527e29fd1477e4" Oct 30 07:28:23.508007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:23.507982 5894 generic.go:356] "Generic (PLEG): container finished" podID="636c5d5e-6e95-45ed-8fd0-d303fc792a8a" containerID="d79b4798091f154d5512844f87afdc743c639adb7c00a41661e1bebfdcbbf7c3" exitCode=0 Oct 30 07:28:23.508122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:23.508070 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" event={"ID":"636c5d5e-6e95-45ed-8fd0-d303fc792a8a","Type":"ContainerDied","Data":"d79b4798091f154d5512844f87afdc743c639adb7c00a41661e1bebfdcbbf7c3"} Oct 30 07:28:24.316494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.316461 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:24.678663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.678621 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:24.699131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.699104 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\") pod \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " Oct 30 07:28:24.699273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.699196 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\" (UniqueName: \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\") pod \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " Oct 30 07:28:24.699320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.699304 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\") pod \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\" (UID: \"636c5d5e-6e95-45ed-8fd0-d303fc792a8a\") " Oct 30 07:28:24.699522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.699503 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca" pod "636c5d5e-6e95-45ed-8fd0-d303fc792a8a" (UID: "636c5d5e-6e95-45ed-8fd0-d303fc792a8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:24.699855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.699831 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:24.701338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.701314 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca" pod "636c5d5e-6e95-45ed-8fd0-d303fc792a8a" (UID: "636c5d5e-6e95-45ed-8fd0-d303fc792a8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:24.701430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.701334 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl" pod "636c5d5e-6e95-45ed-8fd0-d303fc792a8a" (UID: "636c5d5e-6e95-45ed-8fd0-d303fc792a8a"). InnerVolumeSpecName "kube-api-access-894xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:24.801360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.801323 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-894xl\" (UniqueName: \"kubernetes.io/projected/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-kube-api-access-894xl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:24.801360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:24.801356 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/636c5d5e-6e95-45ed-8fd0-d303fc792a8a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:25.525529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:25.525497 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" event={"ID":"636c5d5e-6e95-45ed-8fd0-d303fc792a8a","Type":"ContainerDied","Data":"4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116"} Oct 30 07:28:25.525529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:25.525533 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6" Oct 30 07:28:25.525784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:25.525546 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5f7a440221c34e14a072c77ee6c5365c50784db6d0f39dd962ea7db4457116" Oct 30 07:28:26.029961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:26.029905 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:27.722116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:27.722074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:28.245620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:28.245584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:29.310811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.310752 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:29.701885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.701810 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89"] Oct 30 07:28:29.702401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702388 5894 cpu_manager.go:414] "containerMap: removing container" podUID="636c5d5e-6e95-45ed-8fd0-d303fc792a8a" containerName="revoke-keys" Oct 30 07:28:29.702457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702403 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="636c5d5e-6e95-45ed-8fd0-d303fc792a8a" containerName="revoke-keys" Oct 30 07:28:29.702457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702418 5894 cpu_manager.go:414] "containerMap: removing container" podUID="80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" containerName="revoke-keys" Oct 30 07:28:29.702457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702423 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" containerName="revoke-keys" Oct 30 07:28:29.702558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702502 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="636c5d5e-6e95-45ed-8fd0-d303fc792a8a" containerName="revoke-keys" Oct 30 07:28:29.702558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.702511 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" containerName="revoke-keys" Oct 30 07:28:29.706248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.706231 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.721486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.721465 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:29.725540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.725499 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89"] Oct 30 07:28:29.734224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.734188 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:29.734224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.734205 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:29.734439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.734306 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:29.772829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.772794 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.773003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.772855 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.773077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.773054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdn2\" (UniqueName: \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.874538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.874500 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdn2\" (UniqueName: \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.874767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.874570 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.874767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.874618 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.875216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.875195 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.877090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.877063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:29.898283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:29.898251 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdn2\" (UniqueName: \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\") pod \"maas-api-revoke-sub-d27898f8-8hr89\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:30.020541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:30.020469 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:30.334759 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:30.334711 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872c617e_e333_4f1d_884d_e7fd53e38928.slice/crio-de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d WatchSource:0}: Error finding container de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d: Status 404 returned error can't find the container with id de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d Oct 30 07:28:30.335280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:30.335258 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89"] Oct 30 07:28:30.572713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:30.572667 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" event={"ID":"872c617e-e333-4f1d-884d-e7fd53e38928","Type":"ContainerStarted","Data":"de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d"} Oct 30 07:28:31.042945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:31.042850 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:31.581978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:31.581943 5894 generic.go:356] "Generic (PLEG): container finished" podID="872c617e-e333-4f1d-884d-e7fd53e38928" containerID="9d4c65d9e0aedca2c98f2902ed9bb9293504dc0022aae8fbf6408cb04e8af93e" exitCode=0 Oct 30 07:28:31.582456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:31.582030 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" event={"ID":"872c617e-e333-4f1d-884d-e7fd53e38928","Type":"ContainerDied","Data":"9d4c65d9e0aedca2c98f2902ed9bb9293504dc0022aae8fbf6408cb04e8af93e"} Oct 30 07:28:32.739745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.739718 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:32.742369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.742336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:32.813307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.812093 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\") pod \"872c617e-e333-4f1d-884d-e7fd53e38928\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " Oct 30 07:28:32.813307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.812152 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\" (UniqueName: \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\") pod \"872c617e-e333-4f1d-884d-e7fd53e38928\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " Oct 30 07:28:32.813307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.812356 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\") pod \"872c617e-e333-4f1d-884d-e7fd53e38928\" (UID: \"872c617e-e333-4f1d-884d-e7fd53e38928\") " Oct 30 07:28:32.818956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.818919 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca" pod "872c617e-e333-4f1d-884d-e7fd53e38928" (UID: "872c617e-e333-4f1d-884d-e7fd53e38928"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:32.821742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.821706 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca" pod "872c617e-e333-4f1d-884d-e7fd53e38928" (UID: "872c617e-e333-4f1d-884d-e7fd53e38928"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:32.821937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.821875 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2" pod "872c617e-e333-4f1d-884d-e7fd53e38928" (UID: "872c617e-e333-4f1d-884d-e7fd53e38928"). InnerVolumeSpecName "kube-api-access-chdn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:32.914482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.914403 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:32.914482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.914431 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdn2\" (UniqueName: \"kubernetes.io/projected/872c617e-e333-4f1d-884d-e7fd53e38928-kube-api-access-chdn2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:32.914482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:32.914441 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/872c617e-e333-4f1d-884d-e7fd53e38928-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:33.069714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.069655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:33.601151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.601118 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" event={"ID":"872c617e-e333-4f1d-884d-e7fd53e38928","Type":"ContainerDied","Data":"de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d"} Oct 30 07:28:33.601332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.601173 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de71bf6a9043b04fe29d240e0243c3e0718a672ef4d8884aa0df31d695e2246d" Oct 30 07:28:33.601332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.601224 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89" Oct 30 07:28:33.667671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.667622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:33.807673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:33.794276 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:34.463710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:34.463629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:34.635195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:34.635165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:34.865811 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:28:34.864825 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:28:35.140787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:35.140704 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:28:36.369044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.367106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:36.648923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.648839 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw"] Oct 30 07:28:36.660657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.655515 5894 cpu_manager.go:414] "containerMap: removing container" podUID="872c617e-e333-4f1d-884d-e7fd53e38928" containerName="revoke-keys" Oct 30 07:28:36.660657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.655545 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872c617e-e333-4f1d-884d-e7fd53e38928" containerName="revoke-keys" Oct 30 07:28:36.660657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.655708 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="872c617e-e333-4f1d-884d-e7fd53e38928" containerName="revoke-keys" Oct 30 07:28:36.661226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.661200 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw"] Oct 30 07:28:36.661537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.661463 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.715680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.715653 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:36.715991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.715971 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:36.743000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.742968 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:36.743300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.743284 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:36.777670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.776345 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.777670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.776408 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqs5p\" (UniqueName: \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.777670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.776491 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.912550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.882474 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.912550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.887823 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.912550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.903622 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqs5p\" (UniqueName: \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.912550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.903775 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.912550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.908347 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:36.956151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:36.956048 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqs5p\" (UniqueName: \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\") pod \"maas-api-revoke-sub-91472310-pjkgw\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:37.132928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.132885 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:37.292485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.292451 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:37.498027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.497940 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:37.671771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.671737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:37.845300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.845267 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:37.884975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:37.884935 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw"] Oct 30 07:28:38.033664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.030098 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:38.172676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.167882 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:38.252288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.252027 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8"] Oct 30 07:28:38.297223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.291445 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-fbfb50-jsnt8"] Oct 30 07:28:38.448133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.447598 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e5e8000-8140-4220-adaf-d26df078aca1" path="/var/lib/kubelet/pods/4e5e8000-8140-4220-adaf-d26df078aca1/volumes" Oct 30 07:28:38.713742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.711490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:38.783907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.783870 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" event={"ID":"0e059f2c-6409-40cd-ac90-1fa5b247482b","Type":"ContainerStarted","Data":"b0721b496d379806fbe77013a2985c42b4873e11296528c047c04e59b75e9bfe"} Oct 30 07:28:38.784069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.783935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" event={"ID":"0e059f2c-6409-40cd-ac90-1fa5b247482b","Type":"ContainerStarted","Data":"3d76b77fc050b6944d5f7db09aa4e82d565700c44afa797f099f922396339789"} Oct 30 07:28:38.824962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:38.824899 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" podStartSLOduration=2.824884183 podStartE2EDuration="2.824884183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:38.824026689 +0000 UTC m=+1882.514944125" watchObservedRunningTime="2026-10-30 07:28:38.824884183 +0000 UTC m=+1882.515801682" Oct 30 07:28:39.663625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:39.663592 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:39.795086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:39.795053 5894 generic.go:356] "Generic (PLEG): container finished" podID="0e059f2c-6409-40cd-ac90-1fa5b247482b" containerID="b0721b496d379806fbe77013a2985c42b4873e11296528c047c04e59b75e9bfe" exitCode=0 Oct 30 07:28:39.795511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:39.795144 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" event={"ID":"0e059f2c-6409-40cd-ac90-1fa5b247482b","Type":"ContainerDied","Data":"b0721b496d379806fbe77013a2985c42b4873e11296528c047c04e59b75e9bfe"} Oct 30 07:28:39.808152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:39.808123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:40.375621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:40.375589 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:40.953231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:40.953204 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:41.019007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.018976 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\") pod \"0e059f2c-6409-40cd-ac90-1fa5b247482b\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " Oct 30 07:28:41.019230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.019128 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\") pod \"0e059f2c-6409-40cd-ac90-1fa5b247482b\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " Oct 30 07:28:41.019230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.019210 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\" (UniqueName: \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\") pod \"0e059f2c-6409-40cd-ac90-1fa5b247482b\" (UID: \"0e059f2c-6409-40cd-ac90-1fa5b247482b\") " Oct 30 07:28:41.019571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.019539 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca" pod "0e059f2c-6409-40cd-ac90-1fa5b247482b" (UID: "0e059f2c-6409-40cd-ac90-1fa5b247482b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:41.020424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.019842 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:41.021407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.021378 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca" pod "0e059f2c-6409-40cd-ac90-1fa5b247482b" (UID: "0e059f2c-6409-40cd-ac90-1fa5b247482b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:41.021534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.021514 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p" pod "0e059f2c-6409-40cd-ac90-1fa5b247482b" (UID: "0e059f2c-6409-40cd-ac90-1fa5b247482b"). InnerVolumeSpecName "kube-api-access-gqs5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:41.121572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.121489 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e059f2c-6409-40cd-ac90-1fa5b247482b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:41.121572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.121519 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqs5p\" (UniqueName: \"kubernetes.io/projected/0e059f2c-6409-40cd-ac90-1fa5b247482b-kube-api-access-gqs5p\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:41.812751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.812721 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" Oct 30 07:28:41.812930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.812722 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw" event={"ID":"0e059f2c-6409-40cd-ac90-1fa5b247482b","Type":"ContainerDied","Data":"3d76b77fc050b6944d5f7db09aa4e82d565700c44afa797f099f922396339789"} Oct 30 07:28:41.812930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:41.812858 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d76b77fc050b6944d5f7db09aa4e82d565700c44afa797f099f922396339789" Oct 30 07:28:42.036770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:42.036737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:43.758079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:43.758041 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:43.873967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:43.873929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:45.327777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.327740 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:45.627089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.627012 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj"] Oct 30 07:28:45.627588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.627576 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0e059f2c-6409-40cd-ac90-1fa5b247482b" containerName="revoke-keys" Oct 30 07:28:45.627658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.627590 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e059f2c-6409-40cd-ac90-1fa5b247482b" containerName="revoke-keys" Oct 30 07:28:45.627733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.627723 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e059f2c-6409-40cd-ac90-1fa5b247482b" containerName="revoke-keys" Oct 30 07:28:45.631444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.631426 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.646919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.646890 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:45.659339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.659307 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:45.659498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.659345 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:45.659498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.659482 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:45.665082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.665056 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj"] Oct 30 07:28:45.679154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.679126 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9794\" (UniqueName: \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.679285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.679169 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.679454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.679433 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.780629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.780585 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.780852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.780715 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9794\" (UniqueName: \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.780852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.780742 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.781412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.781391 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.783301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.783264 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.804734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.804697 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9794\" (UniqueName: \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\") pod \"maas-api-revoke-sub-1ac73e1d-mb2mj\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:45.945288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:45.945213 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:46.266826 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:46.266792 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda166e2fa_2d80_4533_947a_fc83128a4d6f.slice/crio-f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4 WatchSource:0}: Error finding container f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4: Status 404 returned error can't find the container with id f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4 Oct 30 07:28:46.267760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:46.267733 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj"] Oct 30 07:28:46.850736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:46.850646 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" event={"ID":"a166e2fa-2d80-4533-947a-fc83128a4d6f","Type":"ContainerStarted","Data":"87221f6cfadd74d0d4ae9ebcef7d57761bb4148c1dbf9b46adf9515e628b8ae7"} Oct 30 07:28:46.850736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:46.850706 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" event={"ID":"a166e2fa-2d80-4533-947a-fc83128a4d6f","Type":"ContainerStarted","Data":"f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4"} Oct 30 07:28:46.893081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:46.893025 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" podStartSLOduration=1.893010767 podStartE2EDuration="1.893010767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:28:46.890761211 +0000 UTC m=+1890.581678649" watchObservedRunningTime="2026-10-30 07:28:46.893010767 +0000 UTC m=+1890.583928241" Oct 30 07:28:47.137600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:47.137497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:47.385651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:47.385598 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:47.863174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:47.863136 5894 generic.go:356] "Generic (PLEG): container finished" podID="a166e2fa-2d80-4533-947a-fc83128a4d6f" containerID="87221f6cfadd74d0d4ae9ebcef7d57761bb4148c1dbf9b46adf9515e628b8ae7" exitCode=0 Oct 30 07:28:47.863610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:47.863291 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" event={"ID":"a166e2fa-2d80-4533-947a-fc83128a4d6f","Type":"ContainerDied","Data":"87221f6cfadd74d0d4ae9ebcef7d57761bb4148c1dbf9b46adf9515e628b8ae7"} Oct 30 07:28:48.048623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:48.048590 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:48.275505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:48.274386 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:48.909458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:48.909347 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:49.067327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.067302 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:49.129184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.129094 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\") pod \"a166e2fa-2d80-4533-947a-fc83128a4d6f\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " Oct 30 07:28:49.129377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.129342 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\" (UniqueName: \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\") pod \"a166e2fa-2d80-4533-947a-fc83128a4d6f\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " Oct 30 07:28:49.129437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.129408 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\") pod \"a166e2fa-2d80-4533-947a-fc83128a4d6f\" (UID: \"a166e2fa-2d80-4533-947a-fc83128a4d6f\") " Oct 30 07:28:49.129500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.129454 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca" pod "a166e2fa-2d80-4533-947a-fc83128a4d6f" (UID: "a166e2fa-2d80-4533-947a-fc83128a4d6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:28:49.129989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.129961 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:49.131704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.131674 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794" pod "a166e2fa-2d80-4533-947a-fc83128a4d6f" (UID: "a166e2fa-2d80-4533-947a-fc83128a4d6f"). InnerVolumeSpecName "kube-api-access-h9794". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:28:49.132241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.132225 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca" pod "a166e2fa-2d80-4533-947a-fc83128a4d6f" (UID: "a166e2fa-2d80-4533-947a-fc83128a4d6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:28:49.230955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.230915 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9794\" (UniqueName: \"kubernetes.io/projected/a166e2fa-2d80-4533-947a-fc83128a4d6f-kube-api-access-h9794\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:49.230955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.230951 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a166e2fa-2d80-4533-947a-fc83128a4d6f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:28:49.882470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.882440 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" Oct 30 07:28:49.882707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.882440 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj" event={"ID":"a166e2fa-2d80-4533-947a-fc83128a4d6f","Type":"ContainerDied","Data":"f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4"} Oct 30 07:28:49.882707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:49.882560 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f691f287d2b5e01b94a47a1d1337ff20ab5388dbac30e35cd5d516a8d0e549f4" Oct 30 07:28:50.515309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:50.515266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:51.591829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:51.591794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:51.714988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:51.714952 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:52.338219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:52.338180 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:53.307543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:53.307499 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:53.434712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:53.434677 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:54.287241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:54.287205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:54.401518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:54.401482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:55.801063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:55.801021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:55.980196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:55.980150 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:56.095704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:56.095609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:56.265476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:56.265446 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:56.587362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:56.587311 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:57.082321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.082290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:57.402247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.402156 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:57.957569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.957532 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:57.981535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.981501 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq"] Oct 30 07:28:57.982148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.982132 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a166e2fa-2d80-4533-947a-fc83128a4d6f" containerName="revoke-keys" Oct 30 07:28:57.982206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.982149 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a166e2fa-2d80-4533-947a-fc83128a4d6f" containerName="revoke-keys" Oct 30 07:28:57.982254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:57.982245 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a166e2fa-2d80-4533-947a-fc83128a4d6f" containerName="revoke-keys" Oct 30 07:28:58.009832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.009795 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.025998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.024317 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq"] Oct 30 07:28:58.028125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.026833 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:28:58.040021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.039994 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:28:58.040421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.040301 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:28:58.040648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.040350 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:28:58.086498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.086464 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.086953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.086607 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.086953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.086754 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnq8d\" (UniqueName: \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.187699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.187665 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.187880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.187781 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.187880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.187862 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnq8d\" (UniqueName: \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.188613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.188579 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.190294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.190267 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.211763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.211699 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnq8d\" (UniqueName: \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\") pod \"maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.350536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.350491 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:28:58.700433 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:28:58.700401 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc953ca_195a_4ee1_bd4a_f5d3bc83c19a.slice/crio-369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1 WatchSource:0}: Error finding container 369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1: Status 404 returned error can't find the container with id 369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1 Oct 30 07:28:58.700567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.700516 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq"] Oct 30 07:28:58.985337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:58.985302 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" event={"ID":"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a","Type":"ContainerStarted","Data":"369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1"} Oct 30 07:28:59.879946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:59.879913 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:28:59.994858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:59.994769 5894 generic.go:356] "Generic (PLEG): container finished" podID="1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" containerID="ed10126e6309d370aa6750db67a926608a0b6ccc0a706fefda71778dc0f09649" exitCode=0 Oct 30 07:28:59.995025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:28:59.994855 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" event={"ID":"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a","Type":"ContainerDied","Data":"ed10126e6309d370aa6750db67a926608a0b6ccc0a706fefda71778dc0f09649"} Oct 30 07:29:00.797656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:00.797394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:00.914181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:00.914146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:01.192829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.192787 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:29:01.242871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.241691 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\" (UniqueName: \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\") pod \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " Oct 30 07:29:01.243183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.243159 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\") pod \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " Oct 30 07:29:01.243549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.243497 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\") pod \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\" (UID: \"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a\") " Oct 30 07:29:01.244874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.244844 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca" pod "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" (UID: "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:01.245263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.245212 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:01.250834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.250783 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca" pod "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" (UID: "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:01.250834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.250795 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d" pod "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" (UID: "1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a"). InnerVolumeSpecName "kube-api-access-tnq8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:01.346874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.346762 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:01.346874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:01.346798 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnq8d\" (UniqueName: \"kubernetes.io/projected/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a-kube-api-access-tnq8d\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:02.153149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:02.153051 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:02.180072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:02.180033 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" event={"ID":"1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a","Type":"ContainerDied","Data":"369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1"} Oct 30 07:29:02.180328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:02.180309 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369a01c3694056e6fab78265b729864e8a5e96d806e2b612c8081e8297612de1" Oct 30 07:29:02.180510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:02.180496 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq" Oct 30 07:29:03.285675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:03.285629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:03.408941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:03.408906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:04.119093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:04.119065 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:04.832935 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:29:04.832884 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:29:05.914674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:05.914621 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:07.615271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:07.615234 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:08.728798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:08.728761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:08.858772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:08.858743 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:09.288229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:09.288192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:09.404521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:09.404487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:09.520251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:09.518473 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:11.027800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:11.027764 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:12.650367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:12.650336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:14.320957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:14.320925 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:16.215140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:16.215109 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:17.683574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:17.683528 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:17.927548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:17.927514 5894 scope.go:122] "RemoveContainer" containerID="2289b5c80c8c7f0f0b0ca05eca903745aba0d39ff610a8097e0d7d31f51d3a8d" Oct 30 07:29:17.937283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:17.937212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:17.938219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:17.938190 5894 scope.go:122] "RemoveContainer" containerID="684c79e892b19c91e4b2d527f00ad0d3173e9ef6e0a993874df7f4945bf1e333" Oct 30 07:29:17.949277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:17.949255 5894 scope.go:122] "RemoveContainer" containerID="25ce696ea742a5ddd1ed15c128efb3c9bf2329a1a18ba37764068ad219ac964f" Oct 30 07:29:18.136444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:18.136411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:18.607015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:18.606980 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:18.784875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:18.784825 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:19.683573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:19.683543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:19.884004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:19.883969 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:20.004808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:20.004707 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:20.721136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:20.721098 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:21.703657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:21.687628 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:21.800516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:21.800482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:22.736332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:22.736302 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:22.894202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:22.894169 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:23.379212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:23.379064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:23.993336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:23.993299 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:24.116544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.116501 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:24.141716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.141677 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq"] Oct 30 07:29:24.154052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.154017 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-a83dbd-z4gbq"] Oct 30 07:29:24.230031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.229995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:24.431113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.431077 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" path="/var/lib/kubelet/pods/1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a/volumes" Oct 30 07:29:24.631798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.631759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:24.829671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.829623 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:24.946597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:24.946562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:25.255668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:25.255561 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:29:25.909426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:25.909394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:27.653847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:27.653815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:27.811308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:27.811273 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:28.004830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.004740 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:28.525257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.525227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:28.646503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.646470 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m"] Oct 30 07:29:28.647176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.647149 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" containerName="revoke-keys" Oct 30 07:29:28.647176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.647172 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" containerName="revoke-keys" Oct 30 07:29:28.647350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.647342 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc953ca-195a-4ee1-bd4a-f5d3bc83c19a" containerName="revoke-keys" Oct 30 07:29:28.652483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.652462 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.670968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.670944 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m"] Oct 30 07:29:28.679333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.679308 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:29:28.679333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.679308 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:29:28.684028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.684004 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:29:28.684142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.684120 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:29:28.684545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.684524 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.684686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.684670 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr6dc\" (UniqueName: \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.684759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.684716 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.785465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.785433 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.785726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.785624 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr6dc\" (UniqueName: \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.785824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.785802 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.786357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.786320 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.788398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.788377 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.809684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.809647 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr6dc\" (UniqueName: \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\") pod \"maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:28.966225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:28.966191 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:29.272542 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:29:29.272511 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfaa91fd_06fb_4568_b11c_be9cb0aaec5b.slice/crio-447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab WatchSource:0}: Error finding container 447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab: Status 404 returned error can't find the container with id 447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab Oct 30 07:29:29.273161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:29.273139 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m"] Oct 30 07:29:29.489262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:29.489229 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" event={"ID":"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b","Type":"ContainerStarted","Data":"447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab"} Oct 30 07:29:29.795328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:29.795292 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:29:29.795796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:29.795562 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" containerID="cri-o://334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671" gracePeriod=30 Oct 30 07:29:29.844735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:29.844687 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:30.498903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:30.498822 5894 generic.go:356] "Generic (PLEG): container finished" podID="cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" containerID="25dcf9ccacd73c0065dd3b1cf85c97ed6cd5c397348450a867b07700c2072fc8" exitCode=0 Oct 30 07:29:30.499057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:30.498909 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" event={"ID":"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b","Type":"ContainerDied","Data":"25dcf9ccacd73c0065dd3b1cf85c97ed6cd5c397348450a867b07700c2072fc8"} Oct 30 07:29:31.416670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.414118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:31.542362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.542325 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:31.676034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.675959 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:31.699785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.699734 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:31.742500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.737478 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\") pod \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " Oct 30 07:29:31.742742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.742565 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\") pod \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " Oct 30 07:29:31.742742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.739803 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca" pod "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" (UID: "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:31.742888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.742795 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\" (UniqueName: \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\") pod \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\" (UID: \"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b\") " Oct 30 07:29:31.743387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.743364 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:31.754171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.754042 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca" pod "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" (UID: "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:31.767160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.767122 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc" pod "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" (UID: "cfaa91fd-06fb-4568-b11c-be9cb0aaec5b"). InnerVolumeSpecName "kube-api-access-dr6dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:31.844477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.844445 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr6dc\" (UniqueName: \"kubernetes.io/projected/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-kube-api-access-dr6dc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:31.844477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:31.844474 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:32.268692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.268648 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:32.388323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.388286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:32.516140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.516112 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" Oct 30 07:29:32.516601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.516153 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m" event={"ID":"cfaa91fd-06fb-4568-b11c-be9cb0aaec5b","Type":"ContainerDied","Data":"447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab"} Oct 30 07:29:32.516601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.516197 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447a33796a8127355bba36fa291df6e70a3dbadd6da2ec0bd1bc24be7ce982ab" Oct 30 07:29:32.734691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.734585 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:32.849424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.849390 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487"] Oct 30 07:29:32.849985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.849968 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" containerName="revoke-keys" Oct 30 07:29:32.850062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.849989 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" containerName="revoke-keys" Oct 30 07:29:32.850169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.850158 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" containerName="revoke-keys" Oct 30 07:29:32.853701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.853679 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:32.871514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.871358 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:29:32.871514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.871380 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:29:32.877953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.877891 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:29:32.878505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.878483 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:29:32.879712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.879682 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487"] Oct 30 07:29:32.963975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.963937 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdw7h\" (UniqueName: \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:32.964158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.964084 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:32.964158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:32.964127 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.065867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.065823 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdw7h\" (UniqueName: \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.066061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.065919 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.066061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.065946 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.066763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.066732 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.068493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.068461 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.089789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.089759 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdw7h\" (UniqueName: \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\") pod \"maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.174905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.174870 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:33.218325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.205496 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:33.426954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.426113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:33.515275 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:29:33.515239 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df81ad3_07d1_44c7_856e_125f845e557e.slice/crio-910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503 WatchSource:0}: Error finding container 910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503: Status 404 returned error can't find the container with id 910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503 Oct 30 07:29:33.515410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.515385 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487"] Oct 30 07:29:33.528190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.528164 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" event={"ID":"7df81ad3-07d1-44c7-856e-125f845e557e","Type":"ContainerStarted","Data":"910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503"} Oct 30 07:29:33.730172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:33.730082 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:34.537991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:34.537959 5894 generic.go:356] "Generic (PLEG): container finished" podID="7df81ad3-07d1-44c7-856e-125f845e557e" containerID="036bf5b0cf82b9753eb05b835591a1eae48ccfb1482de4b1c950857ff2d1cb13" exitCode=0 Oct 30 07:29:34.538467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:34.538052 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" event={"ID":"7df81ad3-07d1-44c7-856e-125f845e557e","Type":"ContainerDied","Data":"036bf5b0cf82b9753eb05b835591a1eae48ccfb1482de4b1c950857ff2d1cb13"} Oct 30 07:29:34.833229 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:29:34.833191 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:29:35.074884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.074819 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:29:35.109063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109023 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109153 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109562 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109591 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109669 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109767 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109798 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.109867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109842 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.110027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.109883 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\") pod \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\" (UID: \"2b9facc6-315b-4f14-95d8-e4e3de5928d2\") " Oct 30 07:29:35.111582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.111287 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:35.111582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.111520 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:35.111582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.111575 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:35.111863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.111689 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:35.111863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.111779 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:35.113028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.113005 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "kube-api-access-tkk2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:35.114197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.114047 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:29:35.114394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.114368 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:35.116312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.116285 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy" pod "2b9facc6-315b-4f14-95d8-e4e3de5928d2" (UID: "2b9facc6-315b-4f14-95d8-e4e3de5928d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:29:35.211599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211565 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211596 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211606 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211615 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211623 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211665 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211674 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9facc6-315b-4f14-95d8-e4e3de5928d2-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211683 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkk2f\" (UniqueName: \"kubernetes.io/projected/2b9facc6-315b-4f14-95d8-e4e3de5928d2-kube-api-access-tkk2f\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.211855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.211693 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9facc6-315b-4f14-95d8-e4e3de5928d2-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.547579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.547540 5894 generic.go:356] "Generic (PLEG): container finished" podID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerID="334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671" exitCode=0 Oct 30 07:29:35.548034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.547605 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" Oct 30 07:29:35.548034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.547626 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" event={"ID":"2b9facc6-315b-4f14-95d8-e4e3de5928d2","Type":"ContainerDied","Data":"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671"} Oct 30 07:29:35.548034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.547701 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt" event={"ID":"2b9facc6-315b-4f14-95d8-e4e3de5928d2","Type":"ContainerDied","Data":"8932f5662e18b0e45ad5b891e09c80045e6debdd862fb89415e123f2811a0f86"} Oct 30 07:29:35.548034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.547721 5894 scope.go:122] "RemoveContainer" containerID="334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671" Oct 30 07:29:35.559839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.559817 5894 scope.go:122] "RemoveContainer" containerID="334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671" Oct 30 07:29:35.560250 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:29:35.560232 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671\": container with ID starting with 334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671 not found: ID does not exist" containerID="334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671" Oct 30 07:29:35.560309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.560258 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671"} err="failed to get container status \"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671\": rpc error: code = NotFound desc = could not find container \"334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671\": container with ID starting with 334374831f41971174b381e5697781d8408f1702fda802a322914df2a6170671 not found: ID does not exist" Oct 30 07:29:35.572220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.572190 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:35.623210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.623137 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:29:35.629036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.629001 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-cvdjt"] Oct 30 07:29:35.735684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.735659 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:35.825341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.825249 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\") pod \"7df81ad3-07d1-44c7-856e-125f845e557e\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " Oct 30 07:29:35.825341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.825335 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\") pod \"7df81ad3-07d1-44c7-856e-125f845e557e\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " Oct 30 07:29:35.825557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.825449 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\" (UniqueName: \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\") pod \"7df81ad3-07d1-44c7-856e-125f845e557e\" (UID: \"7df81ad3-07d1-44c7-856e-125f845e557e\") " Oct 30 07:29:35.825766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.825734 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca" pod "7df81ad3-07d1-44c7-856e-125f845e557e" (UID: "7df81ad3-07d1-44c7-856e-125f845e557e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:29:35.826059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.826040 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.827527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.827500 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h" pod "7df81ad3-07d1-44c7-856e-125f845e557e" (UID: "7df81ad3-07d1-44c7-856e-125f845e557e"). InnerVolumeSpecName "kube-api-access-fdw7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:29:35.827599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.827509 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca" pod "7df81ad3-07d1-44c7-856e-125f845e557e" (UID: "7df81ad3-07d1-44c7-856e-125f845e557e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:29:35.926761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.926725 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdw7h\" (UniqueName: \"kubernetes.io/projected/7df81ad3-07d1-44c7-856e-125f845e557e-kube-api-access-fdw7h\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:35.926761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:35.926752 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7df81ad3-07d1-44c7-856e-125f845e557e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:29:36.432296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:36.432259 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" path="/var/lib/kubelet/pods/2b9facc6-315b-4f14-95d8-e4e3de5928d2/volumes" Oct 30 07:29:36.557589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:36.557559 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" Oct 30 07:29:36.557589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:36.557582 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487" event={"ID":"7df81ad3-07d1-44c7-856e-125f845e557e","Type":"ContainerDied","Data":"910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503"} Oct 30 07:29:36.558040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:36.557682 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="910742e99c10ef349b423247b67a712866959b5a4dfcfad53b8142fcf5434503" Oct 30 07:29:37.318319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:37.318282 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:37.454013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:37.453982 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:39.084329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:39.084287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:39.209027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:39.208985 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:39.482428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:39.482341 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:40.623356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:40.623323 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:40.742230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:40.742182 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:41.247357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:41.247326 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:42.905669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:42.905598 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:43.017779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:43.017741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:43.154923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:43.154881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:44.526728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:44.526691 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:44.645831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:44.645792 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:44.809312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:44.809265 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:46.499041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:46.499008 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:46.615180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:46.615148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:47.005690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:47.005627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:47.563599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:47.563545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:48.023452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:48.023305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:48.323700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:48.323669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:50.133721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:50.133690 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:50.875207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:50.875171 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:51.006401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:51.006370 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:52.184515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.184477 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:52.633877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.633822 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:52.750790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.750758 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:52.943232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.943142 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 07:29:52.944274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.944245 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7df81ad3-07d1-44c7-856e-125f845e557e" containerName="revoke-keys" Oct 30 07:29:52.944455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.944441 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7df81ad3-07d1-44c7-856e-125f845e557e" containerName="revoke-keys" Oct 30 07:29:52.944570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.944558 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" Oct 30 07:29:52.944693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.944678 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" Oct 30 07:29:52.945037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.945021 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7df81ad3-07d1-44c7-856e-125f845e557e" containerName="revoke-keys" Oct 30 07:29:52.945143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.945131 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9facc6-315b-4f14-95d8-e4e3de5928d2" containerName="istio-proxy" Oct 30 07:29:52.949910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.949883 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:52.959118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.959084 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 07:29:52.986662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.984052 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-bd38cf\"" Oct 30 07:29:52.986662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:52.984091 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-bd38cf-dockercfg-ngg7t\"" Oct 30 07:29:53.072809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.072776 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bspxt\" (UniqueName: \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.073001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.072973 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.174232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.174197 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspxt\" (UniqueName: \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.174445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.174337 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.175130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.175107 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.198002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.197932 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspxt\" (UniqueName: \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\") pod \"payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.283850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.283816 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.588665 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:29:53.588606 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c913759_de4e_40c6_8169_d3f511319963.slice/crio-81f2c7ab9151b985695cae8d75587bbe730492e6f0b10af9b868963153fdb02e WatchSource:0}: Error finding container 81f2c7ab9151b985695cae8d75587bbe730492e6f0b10af9b868963153fdb02e: Status 404 returned error can't find the container with id 81f2c7ab9151b985695cae8d75587bbe730492e6f0b10af9b868963153fdb02e Oct 30 07:29:53.590421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.590395 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 07:29:53.759441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.759403 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" event={"ID":"3c913759-de4e-40c6-8169-d3f511319963","Type":"ContainerStarted","Data":"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a"} Oct 30 07:29:53.759607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.759474 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:29:53.759607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.759498 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" event={"ID":"3c913759-de4e-40c6-8169-d3f511319963","Type":"ContainerStarted","Data":"81f2c7ab9151b985695cae8d75587bbe730492e6f0b10af9b868963153fdb02e"} Oct 30 07:29:53.804195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.804134 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" podStartSLOduration=1.804118448 podStartE2EDuration="1.804118448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:29:53.800862087 +0000 UTC m=+1957.491779525" watchObservedRunningTime="2026-10-30 07:29:53.804118448 +0000 UTC m=+1957.495035885" Oct 30 07:29:53.989424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:53.989333 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:55.487972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:55.487929 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:56.104342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.104309 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:56.231194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.231155 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:56.262618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.262575 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487"] Oct 30 07:29:56.285438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.285402 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-2ddf97-g9487"] Oct 30 07:29:56.341668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.341514 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:56.431142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.431064 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7df81ad3-07d1-44c7-856e-125f845e557e" path="/var/lib/kubelet/pods/7df81ad3-07d1-44c7-856e-125f845e557e/volumes" Oct 30 07:29:56.796473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:56.796441 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:57.886501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:57.886466 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:57.999177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:57.999138 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:58.116170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:58.116137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:29:59.620051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:29:59.620020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:00.153432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.153397 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:00.158910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.158886 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.166743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.166715 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:00.171084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.171058 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:00.171239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.171176 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.175489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.175462 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:00.175628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.175576 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.179717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.179695 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:00.179816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.179796 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.182074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.182054 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:30:00.182207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.182111 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:30:00.182207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.182168 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:30:00.184662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.184626 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.193949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.193814 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:00.209889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.209538 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:00.219285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.218535 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:00.219285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.219175 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.226801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.226779 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:00.231939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.231910 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:00.238921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.236601 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:00.241216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.241010 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:00.248338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.247143 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.250871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.249854 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:00.256530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.256506 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:00.256801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.256779 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.266426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.264939 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:00.270670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.270597 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.279730 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.279821 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.279852 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z45px\" (UniqueName: \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.279909 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpj6q\" (UniqueName: \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.279989 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ksp2\" (UniqueName: \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.280022 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.280081 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm7lh\" (UniqueName: \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.280128 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.280179 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.281125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.280306 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8xr\" (UniqueName: \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.381269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381235 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.381269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381271 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.381546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381290 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z45px\" (UniqueName: \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.381546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381320 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtp4d\" (UniqueName: \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.381546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381353 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpj6q\" (UniqueName: \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.381546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381500 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ksp2\" (UniqueName: \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.381833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381600 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.381833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381674 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm7lh\" (UniqueName: \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.381833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381709 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.381833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381756 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.382053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.381856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.382113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382047 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.382176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382148 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.382232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382186 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkk9w\" (UniqueName: \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.382291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382230 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.382291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382246 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8xr\" (UniqueName: \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.382291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382256 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.382452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382305 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.382452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382397 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxclp\" (UniqueName: \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.382452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382425 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.382753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.382736 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.405950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.405881 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z45px\" (UniqueName: \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\") pod \"maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.406126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.406060 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ksp2\" (UniqueName: \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\") pod \"maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.406391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.406371 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpj6q\" (UniqueName: \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.410716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.410691 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8xr\" (UniqueName: \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.411037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.411019 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm7lh\" (UniqueName: \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\") pod \"maas-api-key-cleanup-29889090-dfdr8\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.474866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.474832 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:00.484009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.483982 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxclp\" (UniqueName: \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.484148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484040 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtp4d\" (UniqueName: \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.484148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484140 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.484278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484164 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.484278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484211 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.484278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484230 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkk9w\" (UniqueName: \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.484891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484869 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.484980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.484869 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.485057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.485033 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.487736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.487706 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:00.497350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.497328 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:00.507948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.507917 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxclp\" (UniqueName: \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.509137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.509114 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkk9w\" (UniqueName: \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.510539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.510512 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtp4d\" (UniqueName: \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.513167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.513145 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:00.535302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.535272 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:00.554423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.554169 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:00.581914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.581858 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:00.604150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.599781 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:00.817707 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.814588 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda3799d_ef89_4382_a9b1_4517d9a54b54.slice/crio-8417d72650a590bac9eede0c4b2b21ba407858814f75d40c1f97edb2b7fb9752 WatchSource:0}: Error finding container 8417d72650a590bac9eede0c4b2b21ba407858814f75d40c1f97edb2b7fb9752: Status 404 returned error can't find the container with id 8417d72650a590bac9eede0c4b2b21ba407858814f75d40c1f97edb2b7fb9752 Oct 30 07:30:00.817707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.817323 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:00.832665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.832618 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:00.834743 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.834719 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ff548c_7f71_45a8_982e_b21ef91aad78.slice/crio-4d164b66d7d59d6e4e88769f21d60636974d3aaaf9e7e1c75849529beac0a9c2 WatchSource:0}: Error finding container 4d164b66d7d59d6e4e88769f21d60636974d3aaaf9e7e1c75849529beac0a9c2: Status 404 returned error can't find the container with id 4d164b66d7d59d6e4e88769f21d60636974d3aaaf9e7e1c75849529beac0a9c2 Oct 30 07:30:00.895403 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.895374 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628860d8_cb99_45e7_ab41_55935aa5792e.slice/crio-62e3b679a4a515ffdd4543efebaacdfc1fcdb7cd6a2d292684d95cc4e37c3136 WatchSource:0}: Error finding container 62e3b679a4a515ffdd4543efebaacdfc1fcdb7cd6a2d292684d95cc4e37c3136: Status 404 returned error can't find the container with id 62e3b679a4a515ffdd4543efebaacdfc1fcdb7cd6a2d292684d95cc4e37c3136 Oct 30 07:30:00.896548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.896520 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:00.930922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.930892 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:00.976149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.976115 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:00.989463 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.989435 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc99061_8494_4f3f_89ba_3859ff43d6ea.slice/crio-5fef3f45b46dfe8731763ea3a61c33ce0fc77a2190efda5e0a93b23a034d84af WatchSource:0}: Error finding container 5fef3f45b46dfe8731763ea3a61c33ce0fc77a2190efda5e0a93b23a034d84af: Status 404 returned error can't find the container with id 5fef3f45b46dfe8731763ea3a61c33ce0fc77a2190efda5e0a93b23a034d84af Oct 30 07:30:00.989580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:00.989523 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:00.989822 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.989805 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d3744f_99dd_4025_9eb0_b9812a47db52.slice/crio-3ca43934895003c010795770a69f38460099b38d5a6b877d3f86f3407a88c93a WatchSource:0}: Error finding container 3ca43934895003c010795770a69f38460099b38d5a6b877d3f86f3407a88c93a: Status 404 returned error can't find the container with id 3ca43934895003c010795770a69f38460099b38d5a6b877d3f86f3407a88c93a Oct 30 07:30:00.990148 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:00.990125 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bb2bc0_527d_4e94_ac3b_f4d5b1549215.slice/crio-9b9f357938402856701c1b370d0bc793151a4786f815a319482c512fb48474d8 WatchSource:0}: Error finding container 9b9f357938402856701c1b370d0bc793151a4786f815a319482c512fb48474d8: Status 404 returned error can't find the container with id 9b9f357938402856701c1b370d0bc793151a4786f815a319482c512fb48474d8 Oct 30 07:30:01.009165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.009134 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:01.012396 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:01.011757 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1153bc_d6a7_4b5d_aad4_a22ebc09093e.slice/crio-773e588e14a5f771a036bc5f7394d5c033a59248a2859fa65491935d54c2c3af WatchSource:0}: Error finding container 773e588e14a5f771a036bc5f7394d5c033a59248a2859fa65491935d54c2c3af: Status 404 returned error can't find the container with id 773e588e14a5f771a036bc5f7394d5c033a59248a2859fa65491935d54c2c3af Oct 30 07:30:01.080625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.080598 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:01.177258 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:01.177224 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33841f7b_7615_40a7_b77b_96cc3c677cca.slice/crio-94d2888ceacd010a1cd7ed0def8b3979d6ce6d159d4afca7f67036f7fba9c0d6 WatchSource:0}: Error finding container 94d2888ceacd010a1cd7ed0def8b3979d6ce6d159d4afca7f67036f7fba9c0d6: Status 404 returned error can't find the container with id 94d2888ceacd010a1cd7ed0def8b3979d6ce6d159d4afca7f67036f7fba9c0d6 Oct 30 07:30:01.273588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.273555 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:01.320057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.320022 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:01.827989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.827954 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerStarted","Data":"c1ca8136526e4b846a38044a0b851bd379bb1e16b5a161429191a4905552cf79"} Oct 30 07:30:01.828441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.828017 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerStarted","Data":"94d2888ceacd010a1cd7ed0def8b3979d6ce6d159d4afca7f67036f7fba9c0d6"} Oct 30 07:30:01.829669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.829625 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerStarted","Data":"991b7ebe8d66af9edf044d530cd1ba43e5469d5c1a1a34a05a41bb1f2012776f"} Oct 30 07:30:01.829790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.829685 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerStarted","Data":"9b9f357938402856701c1b370d0bc793151a4786f815a319482c512fb48474d8"} Oct 30 07:30:01.831282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.831258 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerStarted","Data":"f4bb06f6edcac154ff8616ac5802a0e3ce81270ff08d38d0ec804e1c470c823d"} Oct 30 07:30:01.831410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.831305 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerStarted","Data":"62e3b679a4a515ffdd4543efebaacdfc1fcdb7cd6a2d292684d95cc4e37c3136"} Oct 30 07:30:01.832887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.832861 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerStarted","Data":"b80879bb83083b9614f696c0b28c7076b9c690d3e603b52f4810a7e8e98f2b2b"} Oct 30 07:30:01.832991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.832906 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerStarted","Data":"5fef3f45b46dfe8731763ea3a61c33ce0fc77a2190efda5e0a93b23a034d84af"} Oct 30 07:30:01.834508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.834455 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerStarted","Data":"0168b0d9a72581bc8307dde9dd838e0515e3a9727142accc047ea39d9935919b"} Oct 30 07:30:01.834626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.834527 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerStarted","Data":"3ca43934895003c010795770a69f38460099b38d5a6b877d3f86f3407a88c93a"} Oct 30 07:30:01.836398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.836376 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerStarted","Data":"3d98fb087e487ec06b6c8c615bd20034aa7408bf63e9ca5e5d00744c46a829ef"} Oct 30 07:30:01.836507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.836417 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerStarted","Data":"8417d72650a590bac9eede0c4b2b21ba407858814f75d40c1f97edb2b7fb9752"} Oct 30 07:30:01.838062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.838039 5894 generic.go:356] "Generic (PLEG): container finished" podID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerID="dd68f7ad06caaf2aa9d6152f50ae06865eacb1d518ab3d18dc6fc05302df9372" exitCode=6 Oct 30 07:30:01.838156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.838105 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" event={"ID":"76ff548c-7f71-45a8-982e-b21ef91aad78","Type":"ContainerDied","Data":"dd68f7ad06caaf2aa9d6152f50ae06865eacb1d518ab3d18dc6fc05302df9372"} Oct 30 07:30:01.838156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.838140 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" event={"ID":"76ff548c-7f71-45a8-982e-b21ef91aad78","Type":"ContainerStarted","Data":"4d164b66d7d59d6e4e88769f21d60636974d3aaaf9e7e1c75849529beac0a9c2"} Oct 30 07:30:01.838277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.838210 5894 scope.go:122] "RemoveContainer" containerID="dd68f7ad06caaf2aa9d6152f50ae06865eacb1d518ab3d18dc6fc05302df9372" Oct 30 07:30:01.839862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.839845 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerStarted","Data":"264052161a01cea14e9a813065d6f720918554234eb3f0e8bd355c6db746ceeb"} Oct 30 07:30:01.839956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.839881 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerStarted","Data":"773e588e14a5f771a036bc5f7394d5c033a59248a2859fa65491935d54c2c3af"} Oct 30 07:30:01.856232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.856197 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:01.879785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.877283 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" podStartSLOduration=1.8772605489999998 podStartE2EDuration="1.877260549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.87519735 +0000 UTC m=+1965.566114799" watchObservedRunningTime="2026-10-30 07:30:01.877260549 +0000 UTC m=+1965.568177984" Oct 30 07:30:01.923782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.923712 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" podStartSLOduration=1.923695223 podStartE2EDuration="1.923695223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.921297378 +0000 UTC m=+1965.612214815" watchObservedRunningTime="2026-10-30 07:30:01.923695223 +0000 UTC m=+1965.614612658" Oct 30 07:30:01.969501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:01.969438 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" podStartSLOduration=1.969423274 podStartE2EDuration="1.969423274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:01.968471391 +0000 UTC m=+1965.659388829" watchObservedRunningTime="2026-10-30 07:30:01.969423274 +0000 UTC m=+1965.660340740" Oct 30 07:30:02.033284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.033250 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m"] Oct 30 07:30:02.037581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.037555 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-5bb165-6tk5m"] Oct 30 07:30:02.044613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.044538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:02.073390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.072465 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" podStartSLOduration=2.072444866 podStartE2EDuration="2.072444866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.071153546 +0000 UTC m=+1965.762070984" watchObservedRunningTime="2026-10-30 07:30:02.072444866 +0000 UTC m=+1965.763362302" Oct 30 07:30:02.118613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.118559 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" podStartSLOduration=2.118544987 podStartE2EDuration="2.118544987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.118238102 +0000 UTC m=+1965.809155538" watchObservedRunningTime="2026-10-30 07:30:02.118544987 +0000 UTC m=+1965.809462432" Oct 30 07:30:02.171565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.170588 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" podStartSLOduration=2.170569452 podStartE2EDuration="2.170569452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.16894267 +0000 UTC m=+1965.859860108" watchObservedRunningTime="2026-10-30 07:30:02.170569452 +0000 UTC m=+1965.861486892" Oct 30 07:30:02.177258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.177225 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:02.213886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.213830 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" podStartSLOduration=2.213817681 podStartE2EDuration="2.213817681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:02.212986998 +0000 UTC m=+1965.903904434" watchObservedRunningTime="2026-10-30 07:30:02.213817681 +0000 UTC m=+1965.904735118" Oct 30 07:30:02.244311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.244278 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:30:02.244563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.244524 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" containerID="cri-o://3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34" gracePeriod=30 Oct 30 07:30:02.436849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.436811 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaa91fd-06fb-4568-b11c-be9cb0aaec5b" path="/var/lib/kubelet/pods/cfaa91fd-06fb-4568-b11c-be9cb0aaec5b/volumes" Oct 30 07:30:02.599170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.599084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:02.849945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.849867 5894 generic.go:356] "Generic (PLEG): container finished" podID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerID="c1ca8136526e4b846a38044a0b851bd379bb1e16b5a161429191a4905552cf79" exitCode=60 Oct 30 07:30:02.850347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.849950 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerDied","Data":"c1ca8136526e4b846a38044a0b851bd379bb1e16b5a161429191a4905552cf79"} Oct 30 07:30:02.850347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.850207 5894 scope.go:122] "RemoveContainer" containerID="c1ca8136526e4b846a38044a0b851bd379bb1e16b5a161429191a4905552cf79" Oct 30 07:30:02.851717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.851694 5894 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerID="991b7ebe8d66af9edf044d530cd1ba43e5469d5c1a1a34a05a41bb1f2012776f" exitCode=60 Oct 30 07:30:02.851829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.851767 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerDied","Data":"991b7ebe8d66af9edf044d530cd1ba43e5469d5c1a1a34a05a41bb1f2012776f"} Oct 30 07:30:02.852025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.852011 5894 scope.go:122] "RemoveContainer" containerID="991b7ebe8d66af9edf044d530cd1ba43e5469d5c1a1a34a05a41bb1f2012776f" Oct 30 07:30:02.853625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.853606 5894 generic.go:356] "Generic (PLEG): container finished" podID="628860d8-cb99-45e7-ab41-55935aa5792e" containerID="f4bb06f6edcac154ff8616ac5802a0e3ce81270ff08d38d0ec804e1c470c823d" exitCode=60 Oct 30 07:30:02.853814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.853696 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerDied","Data":"f4bb06f6edcac154ff8616ac5802a0e3ce81270ff08d38d0ec804e1c470c823d"} Oct 30 07:30:02.853944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.853927 5894 scope.go:122] "RemoveContainer" containerID="f4bb06f6edcac154ff8616ac5802a0e3ce81270ff08d38d0ec804e1c470c823d" Oct 30 07:30:02.855508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.855487 5894 generic.go:356] "Generic (PLEG): container finished" podID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerID="b80879bb83083b9614f696c0b28c7076b9c690d3e603b52f4810a7e8e98f2b2b" exitCode=60 Oct 30 07:30:02.855584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.855565 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerDied","Data":"b80879bb83083b9614f696c0b28c7076b9c690d3e603b52f4810a7e8e98f2b2b"} Oct 30 07:30:02.855841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.855824 5894 scope.go:122] "RemoveContainer" containerID="b80879bb83083b9614f696c0b28c7076b9c690d3e603b52f4810a7e8e98f2b2b" Oct 30 07:30:02.857297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.857273 5894 generic.go:356] "Generic (PLEG): container finished" podID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerID="0168b0d9a72581bc8307dde9dd838e0515e3a9727142accc047ea39d9935919b" exitCode=60 Oct 30 07:30:02.857383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.857348 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerDied","Data":"0168b0d9a72581bc8307dde9dd838e0515e3a9727142accc047ea39d9935919b"} Oct 30 07:30:02.857520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.857505 5894 scope.go:122] "RemoveContainer" containerID="0168b0d9a72581bc8307dde9dd838e0515e3a9727142accc047ea39d9935919b" Oct 30 07:30:02.859095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.859078 5894 generic.go:356] "Generic (PLEG): container finished" podID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerID="3d98fb087e487ec06b6c8c615bd20034aa7408bf63e9ca5e5d00744c46a829ef" exitCode=60 Oct 30 07:30:02.859173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.859134 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerDied","Data":"3d98fb087e487ec06b6c8c615bd20034aa7408bf63e9ca5e5d00744c46a829ef"} Oct 30 07:30:02.859349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.859335 5894 scope.go:122] "RemoveContainer" containerID="3d98fb087e487ec06b6c8c615bd20034aa7408bf63e9ca5e5d00744c46a829ef" Oct 30 07:30:02.861181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.861159 5894 generic.go:356] "Generic (PLEG): container finished" podID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerID="17b7f2a2e5683cfab81bae33b15c98dee85b7071c7cc9fb6354beee525db7b19" exitCode=6 Oct 30 07:30:02.861854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.861278 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" event={"ID":"76ff548c-7f71-45a8-982e-b21ef91aad78","Type":"ContainerDied","Data":"17b7f2a2e5683cfab81bae33b15c98dee85b7071c7cc9fb6354beee525db7b19"} Oct 30 07:30:02.861854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.861326 5894 scope.go:122] "RemoveContainer" containerID="dd68f7ad06caaf2aa9d6152f50ae06865eacb1d518ab3d18dc6fc05302df9372" Oct 30 07:30:02.864068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.863997 5894 generic.go:356] "Generic (PLEG): container finished" podID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerID="264052161a01cea14e9a813065d6f720918554234eb3f0e8bd355c6db746ceeb" exitCode=60 Oct 30 07:30:02.864068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.864019 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerDied","Data":"264052161a01cea14e9a813065d6f720918554234eb3f0e8bd355c6db746ceeb"} Oct 30 07:30:02.864276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:02.864262 5894 scope.go:122] "RemoveContainer" containerID="264052161a01cea14e9a813065d6f720918554234eb3f0e8bd355c6db746ceeb" Oct 30 07:30:03.019904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.019879 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:03.053426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.053393 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\") pod \"76ff548c-7f71-45a8-982e-b21ef91aad78\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " Oct 30 07:30:03.053568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.053507 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\" (UniqueName: \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\") pod \"76ff548c-7f71-45a8-982e-b21ef91aad78\" (UID: \"76ff548c-7f71-45a8-982e-b21ef91aad78\") " Oct 30 07:30:03.053787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.053764 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca" pod "76ff548c-7f71-45a8-982e-b21ef91aad78" (UID: "76ff548c-7f71-45a8-982e-b21ef91aad78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:03.053934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.053920 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76ff548c-7f71-45a8-982e-b21ef91aad78-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:03.055548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.055527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2" pod "76ff548c-7f71-45a8-982e-b21ef91aad78" (UID: "76ff548c-7f71-45a8-982e-b21ef91aad78"). InnerVolumeSpecName "kube-api-access-9ksp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:03.154793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.154707 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ksp2\" (UniqueName: \"kubernetes.io/projected/76ff548c-7f71-45a8-982e-b21ef91aad78-kube-api-access-9ksp2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:03.877068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.877031 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerStarted","Data":"ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c"} Oct 30 07:30:03.879952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.879919 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerStarted","Data":"042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6"} Oct 30 07:30:03.882058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.882035 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerStarted","Data":"27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd"} Oct 30 07:30:03.884062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.884036 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerStarted","Data":"5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb"} Oct 30 07:30:03.885934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.885913 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerStarted","Data":"b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253"} Oct 30 07:30:03.887911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.887888 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerStarted","Data":"73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4"} Oct 30 07:30:03.889147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.889131 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" Oct 30 07:30:03.889248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.889161 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7" event={"ID":"76ff548c-7f71-45a8-982e-b21ef91aad78","Type":"ContainerDied","Data":"4d164b66d7d59d6e4e88769f21d60636974d3aaaf9e7e1c75849529beac0a9c2"} Oct 30 07:30:03.889248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.889203 5894 scope.go:122] "RemoveContainer" containerID="17b7f2a2e5683cfab81bae33b15c98dee85b7071c7cc9fb6354beee525db7b19" Oct 30 07:30:03.891380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:03.891356 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerStarted","Data":"c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e"} Oct 30 07:30:04.017707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.017669 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:04.027746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.027717 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-5bb165-29889090-rmmg7"] Oct 30 07:30:04.083046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.083009 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:04.431916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.431823 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" path="/var/lib/kubelet/pods/76ff548c-7f71-45a8-982e-b21ef91aad78/volumes" Oct 30 07:30:04.770269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.770186 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 07:30:04.841141 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.841100 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:30:04.901984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.901948 5894 generic.go:356] "Generic (PLEG): container finished" podID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerID="ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c" exitCode=60 Oct 30 07:30:04.902494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.902040 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerDied","Data":"ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c"} Oct 30 07:30:04.902494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.902223 5894 scope.go:122] "RemoveContainer" containerID="c1ca8136526e4b846a38044a0b851bd379bb1e16b5a161429191a4905552cf79" Oct 30 07:30:04.902494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.902347 5894 scope.go:122] "RemoveContainer" containerID="ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c" Oct 30 07:30:04.902710 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.902681 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4_odh-ai-gateway-infra(33841f7b-7615-40a7-b77b-96cc3c677cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" Oct 30 07:30:04.904817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.904796 5894 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerID="042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6" exitCode=60 Oct 30 07:30:04.904907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.904832 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerDied","Data":"042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6"} Oct 30 07:30:04.905239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.905217 5894 scope.go:122] "RemoveContainer" containerID="042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6" Oct 30 07:30:04.905561 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.905536 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828_odh-ai-gateway-infra(e4bb2bc0-527d-4e94-ac3b-f4d5b1549215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" Oct 30 07:30:04.906859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.906839 5894 generic.go:356] "Generic (PLEG): container finished" podID="628860d8-cb99-45e7-ab41-55935aa5792e" containerID="27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd" exitCode=60 Oct 30 07:30:04.906940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.906916 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerDied","Data":"27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd"} Oct 30 07:30:04.907150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.907134 5894 scope.go:122] "RemoveContainer" containerID="27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd" Oct 30 07:30:04.907454 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.907431 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm_odh-ai-gateway-infra(628860d8-cb99-45e7-ab41-55935aa5792e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" Oct 30 07:30:04.909093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.909070 5894 generic.go:356] "Generic (PLEG): container finished" podID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerID="5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb" exitCode=60 Oct 30 07:30:04.909185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.909145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerDied","Data":"5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb"} Oct 30 07:30:04.909367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.909351 5894 scope.go:122] "RemoveContainer" containerID="5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb" Oct 30 07:30:04.909684 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.909664 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-dfdr8_odh-ai-gateway-infra(ddc99061-8494-4f3f-89ba-3859ff43d6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" Oct 30 07:30:04.911219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.911192 5894 generic.go:356] "Generic (PLEG): container finished" podID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerID="b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253" exitCode=60 Oct 30 07:30:04.911310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.911216 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerDied","Data":"b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253"} Oct 30 07:30:04.911520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.911499 5894 scope.go:122] "RemoveContainer" containerID="b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253" Oct 30 07:30:04.911809 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.911789 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks_odh-ai-gateway-infra(89d3744f-99dd-4025-9eb0-b9812a47db52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" Oct 30 07:30:04.913146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.913126 5894 generic.go:356] "Generic (PLEG): container finished" podID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerID="73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4" exitCode=60 Oct 30 07:30:04.913249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.913151 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerDied","Data":"73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4"} Oct 30 07:30:04.913443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.913426 5894 scope.go:122] "RemoveContainer" containerID="73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4" Oct 30 07:30:04.913734 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.913707 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x_odh-ai-gateway-infra(bda3799d-ef89-4382-a9b1-4517d9a54b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" Oct 30 07:30:04.916700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.916672 5894 generic.go:356] "Generic (PLEG): container finished" podID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerID="c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e" exitCode=60 Oct 30 07:30:04.916831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.916776 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerDied","Data":"c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e"} Oct 30 07:30:04.917075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.917057 5894 scope.go:122] "RemoveContainer" containerID="c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e" Oct 30 07:30:04.917355 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:04.917338 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p_odh-ai-gateway-infra(2c1153bc-d6a7-4b5d-aad4-a22ebc09093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" Oct 30 07:30:04.918642 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.918610 5894 scope.go:122] "RemoveContainer" containerID="991b7ebe8d66af9edf044d530cd1ba43e5469d5c1a1a34a05a41bb1f2012776f" Oct 30 07:30:04.938147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.938120 5894 scope.go:122] "RemoveContainer" containerID="f4bb06f6edcac154ff8616ac5802a0e3ce81270ff08d38d0ec804e1c470c823d" Oct 30 07:30:04.953245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.953226 5894 scope.go:122] "RemoveContainer" containerID="b80879bb83083b9614f696c0b28c7076b9c690d3e603b52f4810a7e8e98f2b2b" Oct 30 07:30:04.968076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.968045 5894 scope.go:122] "RemoveContainer" containerID="0168b0d9a72581bc8307dde9dd838e0515e3a9727142accc047ea39d9935919b" Oct 30 07:30:04.987997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:04.987976 5894 scope.go:122] "RemoveContainer" containerID="3d98fb087e487ec06b6c8c615bd20034aa7408bf63e9ca5e5d00744c46a829ef" Oct 30 07:30:05.003568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.003540 5894 scope.go:122] "RemoveContainer" containerID="264052161a01cea14e9a813065d6f720918554234eb3f0e8bd355c6db746ceeb" Oct 30 07:30:05.131425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.131395 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:05.317652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.317609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:05.419573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.419484 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:30:05.584075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.584038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:05.863568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.863532 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:05.928406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.928374 5894 scope.go:122] "RemoveContainer" containerID="ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c" Oct 30 07:30:05.928884 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.928711 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4_odh-ai-gateway-infra(33841f7b-7615-40a7-b77b-96cc3c677cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" Oct 30 07:30:05.930246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.930227 5894 scope.go:122] "RemoveContainer" containerID="042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6" Oct 30 07:30:05.930540 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.930509 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828_odh-ai-gateway-infra(e4bb2bc0-527d-4e94-ac3b-f4d5b1549215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" Oct 30 07:30:05.931824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.931803 5894 scope.go:122] "RemoveContainer" containerID="27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd" Oct 30 07:30:05.932091 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.932071 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm_odh-ai-gateway-infra(628860d8-cb99-45e7-ab41-55935aa5792e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" Oct 30 07:30:05.933389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.933372 5894 scope.go:122] "RemoveContainer" containerID="5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb" Oct 30 07:30:05.933663 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.933617 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-dfdr8_odh-ai-gateway-infra(ddc99061-8494-4f3f-89ba-3859ff43d6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" Oct 30 07:30:05.935005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.934988 5894 scope.go:122] "RemoveContainer" containerID="b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253" Oct 30 07:30:05.935220 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.935198 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks_odh-ai-gateway-infra(89d3744f-99dd-4025-9eb0-b9812a47db52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" Oct 30 07:30:05.936400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.936382 5894 scope.go:122] "RemoveContainer" containerID="73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4" Oct 30 07:30:05.936627 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.936602 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x_odh-ai-gateway-infra(bda3799d-ef89-4382-a9b1-4517d9a54b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" Oct 30 07:30:05.938223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:05.938206 5894 scope.go:122] "RemoveContainer" containerID="c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e" Oct 30 07:30:05.938384 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:05.938368 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p_odh-ai-gateway-infra(2c1153bc-d6a7-4b5d-aad4-a22ebc09093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" Oct 30 07:30:07.519694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.519583 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:30:07.650907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.650869 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.650931 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.650990 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651032 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651084 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651117 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651143 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651177 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651246 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:07.651451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651376 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:07.651644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651609 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\") pod \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\" (UID: \"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0\") " Oct 30 07:30:07.651728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651681 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:07.651886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651861 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:07.651955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.651873 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:07.652323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.652299 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.652437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.652326 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.652437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.652342 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.652437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.652355 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.652437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.652368 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.653529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.653502 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "kube-api-access-vflsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:07.653613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.653541 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:30:07.653613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.653551 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:30:07.653755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.653660 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token" pod "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" (UID: "a894a0ef-dc4a-438a-95cd-3cfa4cebadd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:07.697248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.697210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:07.756025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.755990 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vflsz\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-kube-api-access-vflsz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.756025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.756027 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.756319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.756044 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.756319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.756056 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:07.957264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.957230 5894 generic.go:356] "Generic (PLEG): container finished" podID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerID="3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34" exitCode=0 Oct 30 07:30:07.957434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.957295 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" Oct 30 07:30:07.957434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.957310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" event={"ID":"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0","Type":"ContainerDied","Data":"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34"} Oct 30 07:30:07.957434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.957372 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq" event={"ID":"a894a0ef-dc4a-438a-95cd-3cfa4cebadd0","Type":"ContainerDied","Data":"ac5fd5bbded235ee7a6f32867472c222d143313563673c00e38fd75bf3427149"} Oct 30 07:30:07.957434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.957396 5894 scope.go:122] "RemoveContainer" containerID="3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34" Oct 30 07:30:07.968906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.968888 5894 scope.go:122] "RemoveContainer" containerID="3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34" Oct 30 07:30:07.969194 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:07.969173 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34\": container with ID starting with 3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34 not found: ID does not exist" containerID="3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34" Oct 30 07:30:07.969262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:07.969203 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34"} err="failed to get container status \"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34\": rpc error: code = NotFound desc = could not find container \"3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34\": container with ID starting with 3ee76a7ac36eda076cd8b323ecaca3e570bded3ce066d630cf6041324cc98a34 not found: ID does not exist" Oct 30 07:30:08.012170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:08.012137 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:30:08.024218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:08.024190 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-gr7hq"] Oct 30 07:30:08.430098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:08.430069 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:08.432155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:08.432131 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" path="/var/lib/kubelet/pods/a894a0ef-dc4a-438a-95cd-3cfa4cebadd0/volumes" Oct 30 07:30:09.150988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:09.150955 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:09.268708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:09.268667 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:09.551032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:09.550966 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:09.747436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:09.747402 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:10.898565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:10.898533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:11.407400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:11.407350 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:11.547604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:11.535465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:11.676729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:11.676512 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:11.861910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:11.861871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:12.142225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.142193 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:30:12.143035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143013 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" Oct 30 07:30:12.143035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143036 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" Oct 30 07:30:12.143193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143058 5894 cpu_manager.go:414] "containerMap: removing container" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:12.143193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143065 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:12.143296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143269 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:12.143296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143290 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:12.143391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.143310 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a894a0ef-dc4a-438a-95cd-3cfa4cebadd0" containerName="istio-proxy" Oct 30 07:30:12.147339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.147316 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.168732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.168703 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:30:12.177032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.177006 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"auth-test-model-2e8b2a-kserve-self-signed-certs\"" Oct 30 07:30:12.177226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.177010 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"default-dockercfg-4f77b\"" Oct 30 07:30:12.178873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.178851 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"kube-root-ca.crt\"" Oct 30 07:30:12.178991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.178893 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"openshift-service-ca.crt\"" Oct 30 07:30:12.205365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.205320 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:12.249961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.249925 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.250163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.249968 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.250163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.250105 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gq79\" (UniqueName: \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.250163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.250138 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.250338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.250202 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.250338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.250231 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.351733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.351690 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.351948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.351833 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.351948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.351876 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.351986 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gq79\" (UniqueName: \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.352027 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.352083 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.352333 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.352368 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.352908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.352807 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.355205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.355137 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.355783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.355760 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.377309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.377273 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gq79\" (UniqueName: \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\") pod \"auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.472620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.471860 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:12.850406 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:12.850288 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8160f7c_ed30_4011_865d_d717c367c9fb.slice/crio-67e4edace722cc209a1a5c34b16b79f6f9995695add5a98caab0f56e4df38266 WatchSource:0}: Error finding container 67e4edace722cc209a1a5c34b16b79f6f9995695add5a98caab0f56e4df38266: Status 404 returned error can't find the container with id 67e4edace722cc209a1a5c34b16b79f6f9995695add5a98caab0f56e4df38266 Oct 30 07:30:12.852726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:12.852696 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:30:13.021346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.021306 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" event={"ID":"b8160f7c-ed30-4011-865d-d717c367c9fb","Type":"ContainerStarted","Data":"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767"} Oct 30 07:30:13.021555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.021355 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" event={"ID":"b8160f7c-ed30-4011-865d-d717c367c9fb","Type":"ContainerStarted","Data":"67e4edace722cc209a1a5c34b16b79f6f9995695add5a98caab0f56e4df38266"} Oct 30 07:30:13.065222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.065155 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" podStartSLOduration=1.065136277 podStartE2EDuration="1.065136277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:13.063566819 +0000 UTC m=+1976.754484247" watchObservedRunningTime="2026-10-30 07:30:13.065136277 +0000 UTC m=+1976.756053715" Oct 30 07:30:13.474440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.474402 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:13.492517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.492484 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:13.878665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.878615 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn"] Oct 30 07:30:13.879290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.879271 5894 cpu_manager.go:414] "containerMap: removing container" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:13.879290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.879288 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ff548c-7f71-45a8-982e-b21ef91aad78" containerName="cleanup" Oct 30 07:30:13.883191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.883159 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:13.898437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.898404 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:30:13.904507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.904431 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn"] Oct 30 07:30:13.976000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.975968 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:13.976168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.976017 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hn9z\" (UniqueName: \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:13.976227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:13.976194 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.077709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.077664 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.077872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.077729 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hn9z\" (UniqueName: \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.077872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.077849 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.078459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.078432 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.080259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.080236 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.101919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.101880 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hn9z\" (UniqueName: \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\") pod \"maas-api-revoke-sub-88885886-bd9gn\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.205770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.205560 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:14.423953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.423918 5894 scope.go:122] "RemoveContainer" containerID="27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd" Oct 30 07:30:14.511209 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:14.511179 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf455f56_b4ac_451a_a3e7_7ff020c6f3a8.slice/crio-c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45 WatchSource:0}: Error finding container c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45: Status 404 returned error can't find the container with id c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45 Oct 30 07:30:14.512284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:14.512264 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn"] Oct 30 07:30:15.039120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.039021 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerStarted","Data":"c8e7ffe3ecf05bccc4af2784029b03ec190a0e7b38396e96023a143f93137f43"} Oct 30 07:30:15.040844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.040808 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" event={"ID":"df455f56-b4ac-451a-a3e7-7ff020c6f3a8","Type":"ContainerStarted","Data":"fa7ac175d3d7dc6e89fca35d97bf32cb1eb65cbc7ca6972b8b1b4f126a826682"} Oct 30 07:30:15.041011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.040862 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" event={"ID":"df455f56-b4ac-451a-a3e7-7ff020c6f3a8","Type":"ContainerStarted","Data":"c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45"} Oct 30 07:30:15.078414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.078373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:15.130569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.130518 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" podStartSLOduration=2.130500448 podStartE2EDuration="2.130500448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:15.126811139 +0000 UTC m=+1978.817728575" watchObservedRunningTime="2026-10-30 07:30:15.130500448 +0000 UTC m=+1978.821417889" Oct 30 07:30:15.204517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.204478 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:15.416720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.416684 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:15.424075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:15.424050 5894 scope.go:122] "RemoveContainer" containerID="042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6" Oct 30 07:30:16.050834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.050799 5894 generic.go:356] "Generic (PLEG): container finished" podID="df455f56-b4ac-451a-a3e7-7ff020c6f3a8" containerID="fa7ac175d3d7dc6e89fca35d97bf32cb1eb65cbc7ca6972b8b1b4f126a826682" exitCode=0 Oct 30 07:30:16.051254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.050885 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" event={"ID":"df455f56-b4ac-451a-a3e7-7ff020c6f3a8","Type":"ContainerDied","Data":"fa7ac175d3d7dc6e89fca35d97bf32cb1eb65cbc7ca6972b8b1b4f126a826682"} Oct 30 07:30:16.052902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.052871 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerStarted","Data":"881f6f51a606df328ae56cce86da2f5705bae12b608a9abe531d324c61fb2dab"} Oct 30 07:30:16.054999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.054975 5894 generic.go:356] "Generic (PLEG): container finished" podID="628860d8-cb99-45e7-ab41-55935aa5792e" containerID="c8e7ffe3ecf05bccc4af2784029b03ec190a0e7b38396e96023a143f93137f43" exitCode=60 Oct 30 07:30:16.055147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.055044 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerDied","Data":"c8e7ffe3ecf05bccc4af2784029b03ec190a0e7b38396e96023a143f93137f43"} Oct 30 07:30:16.055147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.055099 5894 scope.go:122] "RemoveContainer" containerID="27009982176b6276fcd874d9e82cf0046cfbf96b81c0a3818512265958003afd" Oct 30 07:30:16.055464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.055445 5894 scope.go:122] "RemoveContainer" containerID="c8e7ffe3ecf05bccc4af2784029b03ec190a0e7b38396e96023a143f93137f43" Oct 30 07:30:16.055814 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:16.055785 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm_odh-ai-gateway-infra(628860d8-cb99-45e7-ab41-55935aa5792e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" Oct 30 07:30:16.095829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.095794 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:16.429045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.429011 5894 scope.go:122] "RemoveContainer" containerID="5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb" Oct 30 07:30:16.429232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.429145 5894 scope.go:122] "RemoveContainer" containerID="73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4" Oct 30 07:30:16.797457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.797415 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:16.910937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:16.910905 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:17.121397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.121354 5894 generic.go:356] "Generic (PLEG): container finished" podID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerID="881f6f51a606df328ae56cce86da2f5705bae12b608a9abe531d324c61fb2dab" exitCode=60 Oct 30 07:30:17.121963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.121456 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerDied","Data":"881f6f51a606df328ae56cce86da2f5705bae12b608a9abe531d324c61fb2dab"} Oct 30 07:30:17.121963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.121517 5894 scope.go:122] "RemoveContainer" containerID="042ab6c8dbaceb3fc509d476aa107c1091a04214a26f9688ce7392a2228958f6" Oct 30 07:30:17.121963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.121920 5894 scope.go:122] "RemoveContainer" containerID="881f6f51a606df328ae56cce86da2f5705bae12b608a9abe531d324c61fb2dab" Oct 30 07:30:17.122261 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:17.122235 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828_odh-ai-gateway-infra(e4bb2bc0-527d-4e94-ac3b-f4d5b1549215)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" Oct 30 07:30:17.131968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.131739 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerStarted","Data":"a6c3c2011cfe6e934d32c3a63a84c1ec2c1e432af536e60b5e4c75c557931236"} Oct 30 07:30:17.137115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.136262 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerStarted","Data":"e08475aeb0a450e0bd667462a8df19331d5ef3f311c456408c2c2e5c27b6e6b3"} Oct 30 07:30:17.140023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.139998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:17.214117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.213935 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:17.385618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.385589 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:17.423268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.423231 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\") pod \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " Oct 30 07:30:17.423407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.423308 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\" (UniqueName: \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\") pod \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " Oct 30 07:30:17.423624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.423588 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca" pod "df455f56-b4ac-451a-a3e7-7ff020c6f3a8" (UID: "df455f56-b4ac-451a-a3e7-7ff020c6f3a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:17.423835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.423794 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\") pod \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\" (UID: \"df455f56-b4ac-451a-a3e7-7ff020c6f3a8\") " Oct 30 07:30:17.424753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.424119 5894 scope.go:122] "RemoveContainer" containerID="b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253" Oct 30 07:30:17.425346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.425314 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:17.426010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.425982 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z" pod "df455f56-b4ac-451a-a3e7-7ff020c6f3a8" (UID: "df455f56-b4ac-451a-a3e7-7ff020c6f3a8"). InnerVolumeSpecName "kube-api-access-8hn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:17.426694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.426674 5894 scope.go:122] "RemoveContainer" containerID="c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e" Oct 30 07:30:17.428300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.427559 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca" pod "df455f56-b4ac-451a-a3e7-7ff020c6f3a8" (UID: "df455f56-b4ac-451a-a3e7-7ff020c6f3a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:30:17.466546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.466133 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:17.525952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.525864 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\") pod \"628860d8-cb99-45e7-ab41-55935aa5792e\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " Oct 30 07:30:17.526115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.525967 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\" (UniqueName: \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\") pod \"628860d8-cb99-45e7-ab41-55935aa5792e\" (UID: \"628860d8-cb99-45e7-ab41-55935aa5792e\") " Oct 30 07:30:17.526288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.526264 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca" pod "628860d8-cb99-45e7-ab41-55935aa5792e" (UID: "628860d8-cb99-45e7-ab41-55935aa5792e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:17.528332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.528310 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628860d8-cb99-45e7-ab41-55935aa5792e-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:17.528648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.528611 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hn9z\" (UniqueName: \"kubernetes.io/projected/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-kube-api-access-8hn9z\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:17.528648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.528653 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df455f56-b4ac-451a-a3e7-7ff020c6f3a8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:17.533528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.533495 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px" pod "628860d8-cb99-45e7-ab41-55935aa5792e" (UID: "628860d8-cb99-45e7-ab41-55935aa5792e"). InnerVolumeSpecName "kube-api-access-z45px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:17.629913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:17.629872 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z45px\" (UniqueName: \"kubernetes.io/projected/628860d8-cb99-45e7-ab41-55935aa5792e-kube-api-access-z45px\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:18.014814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.014785 5894 scope.go:122] "RemoveContainer" containerID="cff64083dcfc52fc0053a56dc8775f6d640bd22aac0dfdaa7e9ffa2d90d32444" Oct 30 07:30:18.025601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.025572 5894 scope.go:122] "RemoveContainer" containerID="e8fbc22b7923235c6192443d5b20f5ffc73bb9c6ef743773c658d592b3ed4fef" Oct 30 07:30:18.145700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.145662 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerStarted","Data":"0ad50da7641ddcf0fa14ec39e5fe257923c0e95b112a6c2194c83d31cfafef23"} Oct 30 07:30:18.147340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.147316 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" Oct 30 07:30:18.147340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.147330 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn" event={"ID":"df455f56-b4ac-451a-a3e7-7ff020c6f3a8","Type":"ContainerDied","Data":"c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45"} Oct 30 07:30:18.147515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.147369 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c33418b26f1c56bfbc5090cee07aa58b7e4e40aa5c7d3901aacda4a2f2fd45" Oct 30 07:30:18.150888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.150861 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" event={"ID":"628860d8-cb99-45e7-ab41-55935aa5792e","Type":"ContainerDied","Data":"62e3b679a4a515ffdd4543efebaacdfc1fcdb7cd6a2d292684d95cc4e37c3136"} Oct 30 07:30:18.150888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.150883 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm" Oct 30 07:30:18.151047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.150918 5894 scope.go:122] "RemoveContainer" containerID="c8e7ffe3ecf05bccc4af2784029b03ec190a0e7b38396e96023a143f93137f43" Oct 30 07:30:18.153803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.153781 5894 generic.go:356] "Generic (PLEG): container finished" podID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerID="a6c3c2011cfe6e934d32c3a63a84c1ec2c1e432af536e60b5e4c75c557931236" exitCode=60 Oct 30 07:30:18.153922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.153827 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerDied","Data":"a6c3c2011cfe6e934d32c3a63a84c1ec2c1e432af536e60b5e4c75c557931236"} Oct 30 07:30:18.154281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.154262 5894 scope.go:122] "RemoveContainer" containerID="a6c3c2011cfe6e934d32c3a63a84c1ec2c1e432af536e60b5e4c75c557931236" Oct 30 07:30:18.154938 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:18.154571 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889090-dfdr8_odh-ai-gateway-infra(ddc99061-8494-4f3f-89ba-3859ff43d6ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" Oct 30 07:30:18.157622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.157594 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerStarted","Data":"a4fa97fa08fedd5e89372e7516c60204641afeec89caeb0256aba576afb67e78"} Oct 30 07:30:18.160359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.160336 5894 generic.go:356] "Generic (PLEG): container finished" podID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerID="e08475aeb0a450e0bd667462a8df19331d5ef3f311c456408c2c2e5c27b6e6b3" exitCode=60 Oct 30 07:30:18.160492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.160380 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerDied","Data":"e08475aeb0a450e0bd667462a8df19331d5ef3f311c456408c2c2e5c27b6e6b3"} Oct 30 07:30:18.160804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.160779 5894 scope.go:122] "RemoveContainer" containerID="e08475aeb0a450e0bd667462a8df19331d5ef3f311c456408c2c2e5c27b6e6b3" Oct 30 07:30:18.161078 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:18.161056 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x_odh-ai-gateway-infra(bda3799d-ef89-4382-a9b1-4517d9a54b54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" Oct 30 07:30:18.166109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.166085 5894 scope.go:122] "RemoveContainer" containerID="5eea1fc4a3689874b82d7dd0699dc4c012efb70980f502ff1c488eba0ed4e1cb" Oct 30 07:30:18.187741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.187712 5894 scope.go:122] "RemoveContainer" containerID="73d6002e6c6c70519c0bfea5ded648540fed9b984a275d74fa3fd1b7191196c4" Oct 30 07:30:18.309963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.309932 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:18.314168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.314141 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989104-29889090-5wdrm"] Oct 30 07:30:18.317996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.317970 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:18.339527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.339498 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:18.369861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.369827 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:18.432118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.431800 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" path="/var/lib/kubelet/pods/628860d8-cb99-45e7-ab41-55935aa5792e/volumes" Oct 30 07:30:18.441439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.441407 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\") pod \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " Oct 30 07:30:18.441614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.441490 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\" (UniqueName: \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\") pod \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\" (UID: \"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215\") " Oct 30 07:30:18.441770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.441747 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca" pod "e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" (UID: "e4bb2bc0-527d-4e94-ac3b-f4d5b1549215"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:18.441914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.441901 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:18.443517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.443496 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d" pod "e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" (UID: "e4bb2bc0-527d-4e94-ac3b-f4d5b1549215"). InnerVolumeSpecName "kube-api-access-dtp4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:18.483927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.483902 5894 scope.go:122] "RemoveContainer" containerID="ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c" Oct 30 07:30:18.543291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.543255 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtp4d\" (UniqueName: \"kubernetes.io/projected/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215-kube-api-access-dtp4d\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:18.887682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:18.887615 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:19.169916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.169877 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerStarted","Data":"08ed36ee139928ce4c19c18ccaf90244e4798bcb1965b9e0fec04a9d000a6d52"} Oct 30 07:30:19.171335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.171311 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" event={"ID":"e4bb2bc0-527d-4e94-ac3b-f4d5b1549215","Type":"ContainerDied","Data":"9b9f357938402856701c1b370d0bc793151a4786f815a319482c512fb48474d8"} Oct 30 07:30:19.171456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.171363 5894 scope.go:122] "RemoveContainer" containerID="881f6f51a606df328ae56cce86da2f5705bae12b608a9abe531d324c61fb2dab" Oct 30 07:30:19.171456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.171387 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828" Oct 30 07:30:19.176627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.176599 5894 generic.go:356] "Generic (PLEG): container finished" podID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerID="a4fa97fa08fedd5e89372e7516c60204641afeec89caeb0256aba576afb67e78" exitCode=60 Oct 30 07:30:19.176790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.176660 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerDied","Data":"a4fa97fa08fedd5e89372e7516c60204641afeec89caeb0256aba576afb67e78"} Oct 30 07:30:19.177074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.177055 5894 scope.go:122] "RemoveContainer" containerID="a4fa97fa08fedd5e89372e7516c60204641afeec89caeb0256aba576afb67e78" Oct 30 07:30:19.177418 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:19.177386 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks_odh-ai-gateway-infra(89d3744f-99dd-4025-9eb0-b9812a47db52)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" Oct 30 07:30:19.184134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.184117 5894 scope.go:122] "RemoveContainer" containerID="b1d442dd4cf283a9262cfae7d31faf9386f8f97643f33db2ac9d164a4aa5a253" Oct 30 07:30:19.184975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.184954 5894 generic.go:356] "Generic (PLEG): container finished" podID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerID="0ad50da7641ddcf0fa14ec39e5fe257923c0e95b112a6c2194c83d31cfafef23" exitCode=60 Oct 30 07:30:19.185050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.185034 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerDied","Data":"0ad50da7641ddcf0fa14ec39e5fe257923c0e95b112a6c2194c83d31cfafef23"} Oct 30 07:30:19.185359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.185346 5894 scope.go:122] "RemoveContainer" containerID="0ad50da7641ddcf0fa14ec39e5fe257923c0e95b112a6c2194c83d31cfafef23" Oct 30 07:30:19.185611 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:19.185595 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p_odh-ai-gateway-infra(2c1153bc-d6a7-4b5d-aad4-a22ebc09093e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" Oct 30 07:30:19.198741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.198701 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:19.202959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.202934 5894 scope.go:122] "RemoveContainer" containerID="c277af79acc437407d2b8e3634c4b7f4f7b5bdd029310ed89b397d6ab5dcd09e" Oct 30 07:30:19.239357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.239316 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:19.337084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.336862 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:19.340553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.340516 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889090-7n828"] Oct 30 07:30:19.460770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.460734 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:19.464565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.464542 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:19.555071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555037 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\" (UniqueName: \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\") pod \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " Oct 30 07:30:19.555227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555114 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\") pod \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\" (UID: \"ddc99061-8494-4f3f-89ba-3859ff43d6ea\") " Oct 30 07:30:19.555227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555171 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\" (UniqueName: \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\") pod \"bda3799d-ef89-4382-a9b1-4517d9a54b54\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " Oct 30 07:30:19.555227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555194 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\") pod \"bda3799d-ef89-4382-a9b1-4517d9a54b54\" (UID: \"bda3799d-ef89-4382-a9b1-4517d9a54b54\") " Oct 30 07:30:19.555574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555542 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca" pod "ddc99061-8494-4f3f-89ba-3859ff43d6ea" (UID: "ddc99061-8494-4f3f-89ba-3859ff43d6ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:19.555948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555619 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca" pod "bda3799d-ef89-4382-a9b1-4517d9a54b54" (UID: "bda3799d-ef89-4382-a9b1-4517d9a54b54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:19.555948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555812 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddc99061-8494-4f3f-89ba-3859ff43d6ea-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.555948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.555826 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bda3799d-ef89-4382-a9b1-4517d9a54b54-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.557592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.557564 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr" pod "bda3799d-ef89-4382-a9b1-4517d9a54b54" (UID: "bda3799d-ef89-4382-a9b1-4517d9a54b54"). InnerVolumeSpecName "kube-api-access-fr8xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:19.558275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.558251 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh" pod "ddc99061-8494-4f3f-89ba-3859ff43d6ea" (UID: "ddc99061-8494-4f3f-89ba-3859ff43d6ea"). InnerVolumeSpecName "kube-api-access-fm7lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:19.657083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.657040 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm7lh\" (UniqueName: \"kubernetes.io/projected/ddc99061-8494-4f3f-89ba-3859ff43d6ea-kube-api-access-fm7lh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:19.657083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:19.657074 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8xr\" (UniqueName: \"kubernetes.io/projected/bda3799d-ef89-4382-a9b1-4517d9a54b54-kube-api-access-fr8xr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.199141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.195345 5894 generic.go:356] "Generic (PLEG): container finished" podID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerID="08ed36ee139928ce4c19c18ccaf90244e4798bcb1965b9e0fec04a9d000a6d52" exitCode=60 Oct 30 07:30:20.199141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.195509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerDied","Data":"08ed36ee139928ce4c19c18ccaf90244e4798bcb1965b9e0fec04a9d000a6d52"} Oct 30 07:30:20.199141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.195654 5894 scope.go:122] "RemoveContainer" containerID="ddcd639db3955cc82cd03b6a245f809cff2d4249c9ecb0759e5a6199e9636f2c" Oct 30 07:30:20.199141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.196220 5894 scope.go:122] "RemoveContainer" containerID="08ed36ee139928ce4c19c18ccaf90244e4798bcb1965b9e0fec04a9d000a6d52" Oct 30 07:30:20.199141 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:20.196660 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4_odh-ai-gateway-infra(33841f7b-7615-40a7-b77b-96cc3c677cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" Oct 30 07:30:20.202091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.202065 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" event={"ID":"ddc99061-8494-4f3f-89ba-3859ff43d6ea","Type":"ContainerDied","Data":"5fef3f45b46dfe8731763ea3a61c33ce0fc77a2190efda5e0a93b23a034d84af"} Oct 30 07:30:20.202238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.202117 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8" Oct 30 07:30:20.205312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.205292 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" Oct 30 07:30:20.205312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.205305 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x" event={"ID":"bda3799d-ef89-4382-a9b1-4517d9a54b54","Type":"ContainerDied","Data":"8417d72650a590bac9eede0c4b2b21ba407858814f75d40c1f97edb2b7fb9752"} Oct 30 07:30:20.217977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.217951 5894 scope.go:122] "RemoveContainer" containerID="a6c3c2011cfe6e934d32c3a63a84c1ec2c1e432af536e60b5e4c75c557931236" Oct 30 07:30:20.230468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.230431 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:20.231462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.231368 5894 scope.go:122] "RemoveContainer" containerID="e08475aeb0a450e0bd667462a8df19331d5ef3f311c456408c2c2e5c27b6e6b3" Oct 30 07:30:20.295266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.293186 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:20.300670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.299649 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889090-dfdr8"] Oct 30 07:30:20.332968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.332937 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:20.338507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.338473 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889090-mlk8x"] Oct 30 07:30:20.404430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.404400 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:20.421492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.421468 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:20.432311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.432282 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" path="/var/lib/kubelet/pods/bda3799d-ef89-4382-a9b1-4517d9a54b54/volumes" Oct 30 07:30:20.432747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.432733 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" path="/var/lib/kubelet/pods/ddc99061-8494-4f3f-89ba-3859ff43d6ea/volumes" Oct 30 07:30:20.433151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.433138 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" path="/var/lib/kubelet/pods/e4bb2bc0-527d-4e94-ac3b-f4d5b1549215/volumes" Oct 30 07:30:20.468519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.468435 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\" (UniqueName: \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\") pod \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " Oct 30 07:30:20.468519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.468501 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\") pod \"89d3744f-99dd-4025-9eb0-b9812a47db52\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " Oct 30 07:30:20.468799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.468555 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\") pod \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\" (UID: \"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e\") " Oct 30 07:30:20.468799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.468649 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\" (UniqueName: \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\") pod \"89d3744f-99dd-4025-9eb0-b9812a47db52\" (UID: \"89d3744f-99dd-4025-9eb0-b9812a47db52\") " Oct 30 07:30:20.468955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.468920 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca" pod "89d3744f-99dd-4025-9eb0-b9812a47db52" (UID: "89d3744f-99dd-4025-9eb0-b9812a47db52"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:20.469326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.469300 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89d3744f-99dd-4025-9eb0-b9812a47db52-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.469416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.469354 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca" pod "2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" (UID: "2c1153bc-d6a7-4b5d-aad4-a22ebc09093e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:20.471012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.470968 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w" pod "2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" (UID: "2c1153bc-d6a7-4b5d-aad4-a22ebc09093e"). InnerVolumeSpecName "kube-api-access-gkk9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:20.471183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.471158 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp" pod "89d3744f-99dd-4025-9eb0-b9812a47db52" (UID: "89d3744f-99dd-4025-9eb0-b9812a47db52"). InnerVolumeSpecName "kube-api-access-vxclp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:20.570296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.570256 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkk9w\" (UniqueName: \"kubernetes.io/projected/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-kube-api-access-gkk9w\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.570296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.570289 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.570296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.570299 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxclp\" (UniqueName: \"kubernetes.io/projected/89d3744f-99dd-4025-9eb0-b9812a47db52-kube-api-access-vxclp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:20.711260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:20.711224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:21.216931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.216896 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" Oct 30 07:30:21.217414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.216896 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks" event={"ID":"89d3744f-99dd-4025-9eb0-b9812a47db52","Type":"ContainerDied","Data":"3ca43934895003c010795770a69f38460099b38d5a6b877d3f86f3407a88c93a"} Oct 30 07:30:21.217414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.217040 5894 scope.go:122] "RemoveContainer" containerID="a4fa97fa08fedd5e89372e7516c60204641afeec89caeb0256aba576afb67e78" Oct 30 07:30:21.220202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.220178 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" Oct 30 07:30:21.220371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.220198 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p" event={"ID":"2c1153bc-d6a7-4b5d-aad4-a22ebc09093e","Type":"ContainerDied","Data":"773e588e14a5f771a036bc5f7394d5c033a59248a2859fa65491935d54c2c3af"} Oct 30 07:30:21.231548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.231530 5894 scope.go:122] "RemoveContainer" containerID="0ad50da7641ddcf0fa14ec39e5fe257923c0e95b112a6c2194c83d31cfafef23" Oct 30 07:30:21.271330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.271297 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:21.274753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.274725 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889090-s98ks"] Oct 30 07:30:21.376029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.375997 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:21.379950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.379916 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889090-pf59p"] Oct 30 07:30:21.411356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.411328 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:21.484619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.484529 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\" (UniqueName: \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\") pod \"33841f7b-7615-40a7-b77b-96cc3c677cca\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " Oct 30 07:30:21.484810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.484789 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\") pod \"33841f7b-7615-40a7-b77b-96cc3c677cca\" (UID: \"33841f7b-7615-40a7-b77b-96cc3c677cca\") " Oct 30 07:30:21.485206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.485174 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca" pod "33841f7b-7615-40a7-b77b-96cc3c677cca" (UID: "33841f7b-7615-40a7-b77b-96cc3c677cca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:30:21.485323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.485313 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33841f7b-7615-40a7-b77b-96cc3c677cca-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:21.487065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.487041 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q" pod "33841f7b-7615-40a7-b77b-96cc3c677cca" (UID: "33841f7b-7615-40a7-b77b-96cc3c677cca"). InnerVolumeSpecName "kube-api-access-tpj6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:30:21.587138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:21.587105 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpj6q\" (UniqueName: \"kubernetes.io/projected/33841f7b-7615-40a7-b77b-96cc3c677cca-kube-api-access-tpj6q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:30:22.233319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.233287 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" Oct 30 07:30:22.233866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.233318 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4" event={"ID":"33841f7b-7615-40a7-b77b-96cc3c677cca","Type":"ContainerDied","Data":"94d2888ceacd010a1cd7ed0def8b3979d6ce6d159d4afca7f67036f7fba9c0d6"} Oct 30 07:30:22.233866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.233377 5894 scope.go:122] "RemoveContainer" containerID="08ed36ee139928ce4c19c18ccaf90244e4798bcb1965b9e0fec04a9d000a6d52" Oct 30 07:30:22.287044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.287008 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:22.291022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.290993 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889090-p2mb4"] Oct 30 07:30:22.372348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.372315 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:22.445689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.445656 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" path="/var/lib/kubelet/pods/2c1153bc-d6a7-4b5d-aad4-a22ebc09093e/volumes" Oct 30 07:30:22.446167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.446150 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" path="/var/lib/kubelet/pods/33841f7b-7615-40a7-b77b-96cc3c677cca/volumes" Oct 30 07:30:22.446645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.446619 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" path="/var/lib/kubelet/pods/89d3744f-99dd-4025-9eb0-b9812a47db52/volumes" Oct 30 07:30:22.473426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.473386 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:22.486962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:22.486882 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:23.255185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:23.255144 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:30:23.263976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:23.263940 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:24.277423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:24.277387 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:25.373878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:25.373847 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:25.835938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:25.835903 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:25.981346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:25.973713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:26.090079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:26.089972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:27.714163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:27.714064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:28.162655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:28.160096 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:30:28.517552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:28.517434 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:29.250628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:29.250582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:29.376962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:29.376928 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:29.734665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:29.734543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:31.132277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.132242 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:31.246455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.246408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:31.420935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.420828 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 07:30:31.421776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.421749 5894 cpu_manager.go:414] "containerMap: removing container" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.421998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.421974 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422002 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422011 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422022 5894 cpu_manager.go:414] "containerMap: removing container" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422030 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422051 5894 cpu_manager.go:414] "containerMap: removing container" podUID="df455f56-b4ac-451a-a3e7-7ff020c6f3a8" containerName="revoke-keys" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422059 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df455f56-b4ac-451a-a3e7-7ff020c6f3a8" containerName="revoke-keys" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422071 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422078 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422092 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422100 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422111 5894 cpu_manager.go:414] "containerMap: removing container" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422118 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422129 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422136 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422152 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422159 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422167 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422174 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.422184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422190 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422198 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422208 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422214 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422232 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422239 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422262 5894 cpu_manager.go:414] "containerMap: removing container" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422268 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422313 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422320 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422331 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422339 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422503 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422518 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422531 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422541 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422552 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422561 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422572 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422582 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422591 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="df455f56-b4ac-451a-a3e7-7ff020c6f3a8" containerName="revoke-keys" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422600 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422620 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422628 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422653 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422662 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422672 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422681 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:31.423684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.422689 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda3799d-ef89-4382-a9b1-4517d9a54b54" containerName="cleanup" Oct 30 07:30:31.459728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.459694 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.506154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.498842 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-d89593-dockercfg-m4mgf\"" Oct 30 07:30:31.506154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.499010 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-d89593\"" Oct 30 07:30:31.519446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.517494 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 07:30:31.548460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.548420 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phwqr\" (UniqueName: \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.548699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.548629 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.637769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.637737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:31.650050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.650013 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.650269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.650149 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phwqr\" (UniqueName: \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.650832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.650808 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.688512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.688435 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phwqr\" (UniqueName: \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\") pod \"payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:31.795168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:31.795135 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:32.114471 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:32.114433 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc28a87_1187_4860_bad5_ff4673b1f1a3.slice/crio-b9ec47c4902549472afc495a850d0d63a09b54106faa7178a19eaa7a4b08f8b6 WatchSource:0}: Error finding container b9ec47c4902549472afc495a850d0d63a09b54106faa7178a19eaa7a4b08f8b6: Status 404 returned error can't find the container with id b9ec47c4902549472afc495a850d0d63a09b54106faa7178a19eaa7a4b08f8b6 Oct 30 07:30:32.115998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:32.115965 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 07:30:32.330482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:32.330443 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" event={"ID":"2cc28a87-1187-4860-bad5-ff4673b1f1a3","Type":"ContainerStarted","Data":"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd"} Oct 30 07:30:32.330969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:32.330536 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" event={"ID":"2cc28a87-1187-4860-bad5-ff4673b1f1a3","Type":"ContainerStarted","Data":"b9ec47c4902549472afc495a850d0d63a09b54106faa7178a19eaa7a4b08f8b6"} Oct 30 07:30:32.330969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:32.330660 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:32.375204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:32.375084 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" podStartSLOduration=1.3750546799999999 podStartE2EDuration="1.37505468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:32.371514569 +0000 UTC m=+1996.062432011" watchObservedRunningTime="2026-10-30 07:30:32.37505468 +0000 UTC m=+1996.065972120" Oct 30 07:30:33.325437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:33.325384 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:34.899924 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:30:34.899844 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:30:35.118676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.118618 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:35.499925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.499883 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:30:35.500788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500765 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500790 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500820 5894 cpu_manager.go:414] "containerMap: removing container" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500828 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500843 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500851 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500862 5894 cpu_manager.go:414] "containerMap: removing container" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500869 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500886 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:35.500891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500893 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500912 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.500920 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc99061-8494-4f3f-89ba-3859ff43d6ea" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.501105 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="628860d8-cb99-45e7-ab41-55935aa5792e" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.501125 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1153bc-d6a7-4b5d-aad4-a22ebc09093e" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.501136 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4bb2bc0-527d-4e94-ac3b-f4d5b1549215" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.501149 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="33841f7b-7615-40a7-b77b-96cc3c677cca" containerName="cleanup" Oct 30 07:30:35.501174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.501158 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d3744f-99dd-4025-9eb0-b9812a47db52" containerName="cleanup" Oct 30 07:30:35.505219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.505194 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.522963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.522927 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:30:35.530940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.530884 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2761aa\"/\"kube-root-ca.crt\"" Oct 30 07:30:35.530940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.530884 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-2761aa\"/\"openshift-service-ca.crt\"" Oct 30 07:30:35.532621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.532597 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2761aa\"/\"default-dockercfg-d6fp8\"" Oct 30 07:30:35.532621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.532600 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-2761aa\"/\"auth-test-model-2761aa-kserve-self-signed-certs\"" Oct 30 07:30:35.573387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.573298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:35.602047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602010 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.602278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602181 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.602365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602326 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpwp\" (UniqueName: \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.602452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602432 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.602592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602574 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.602689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.602646 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.703794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.703755 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.703830 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.703889 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.703939 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.703968 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpwp\" (UniqueName: \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.704046 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.704356 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.704379 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.704691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.704624 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.706401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.706373 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.706575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.706560 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.766711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.757385 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpwp\" (UniqueName: \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\") pod \"auth-test-model-2761aa-kserve-59b4b76897-bdt5s\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:35.825121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:35.825084 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:36.162887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.162860 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:30:36.163250 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:36.163150 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bdaf9f9_53f9_4cc7_9b1a_de4f5eee35e0.slice/crio-b5639220381d7ef340cc57c07a1de5f19ef930458e00412f45a510b198183782 WatchSource:0}: Error finding container b5639220381d7ef340cc57c07a1de5f19ef930458e00412f45a510b198183782: Status 404 returned error can't find the container with id b5639220381d7ef340cc57c07a1de5f19ef930458e00412f45a510b198183782 Oct 30 07:30:36.367687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.367617 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" event={"ID":"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0","Type":"ContainerStarted","Data":"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c"} Oct 30 07:30:36.367893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.367695 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" event={"ID":"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0","Type":"ContainerStarted","Data":"b5639220381d7ef340cc57c07a1de5f19ef930458e00412f45a510b198183782"} Oct 30 07:30:36.412725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.412601 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" podStartSLOduration=1.412586827 podStartE2EDuration="1.412586827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:36.409129439 +0000 UTC m=+2000.100046876" watchObservedRunningTime="2026-10-30 07:30:36.412586827 +0000 UTC m=+2000.103504281" Oct 30 07:30:36.666693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.666542 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:36.826140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:36.826101 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:37.004295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:37.000889 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:37.130161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:37.130064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:38.222256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:38.222159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:38.842573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:38.842539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:40.412884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:40.412846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:42.001302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:42.001261 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:43.343711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:43.343681 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 07:30:43.578037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:43.577998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:45.247914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:45.247879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:45.825218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:45.825170 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:45.838263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:45.838234 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:46.459091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:46.459061 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:30:46.841958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:46.841925 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:48.620448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:48.620399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:50.313880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:50.313827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:51.855835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:51.855798 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:51.983415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:51.983373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:52.173598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.173504 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:52.794273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.794237 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:30:52.799091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.799064 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.830672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.830625 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:30:52.903958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.903916 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904039 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904123 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904145 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904225 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904279 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904301 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnhm\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904415 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:52.904894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:52.904494 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.005814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.005772 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.005885 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.005914 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.005948 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006012 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006045 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnhm\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006092 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006160 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006210 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006324 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006349 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006666 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.006936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.006916 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.007030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.007007 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.009191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.009164 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.009342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.009302 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.032029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.031993 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.032220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.032086 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnhm\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\") pod \"e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.115697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.115567 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:53.374748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.374658 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:30:53.379711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.379684 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.395281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.395250 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"default-dockercfg-mfvkp\"" Oct 30 07:30:53.399451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.399392 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:30:53.406959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.406931 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"openshift-service-ca.crt\"" Oct 30 07:30:53.407123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.406983 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"auth-test-model-4eff1f-kserve-self-signed-certs\"" Oct 30 07:30:53.407123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.407053 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-4eff1f\"/\"kube-root-ca.crt\"" Oct 30 07:30:53.412066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412035 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.412211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412089 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.412288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412204 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftqt\" (UniqueName: \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.412352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412335 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.412411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412373 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.412411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.412404 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.438131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.438085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:53.480716 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:30:53.480625 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f1bc3b_6c56_4146_b73e_0cf5dda18116.slice/crio-5dda0f9bea8cb7b74283cd5f735f4c55971ca32e85a2ef68283d329225a306aa WatchSource:0}: Error finding container 5dda0f9bea8cb7b74283cd5f735f4c55971ca32e85a2ef68283d329225a306aa: Status 404 returned error can't find the container with id 5dda0f9bea8cb7b74283cd5f735f4c55971ca32e85a2ef68283d329225a306aa Oct 30 07:30:53.482219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.482192 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:30:53.484266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.484234 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:53.484341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.484304 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:53.484385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.484339 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:30:53.513957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.513930 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.513989 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514042 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftqt\" (UniqueName: \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514135 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514171 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514201 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514368 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.514864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514835 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.515004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.514967 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.516464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.516438 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.516902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.516882 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.539538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.539507 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftqt\" (UniqueName: \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\") pod \"auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.570051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.570008 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" event={"ID":"09f1bc3b-6c56-4146-b73e-0cf5dda18116","Type":"ContainerStarted","Data":"03fbddea6e07473661f7fc8150314cc0856a8a1fc3dec35031f3a100b29e778c"} Oct 30 07:30:53.570238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.570076 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" event={"ID":"09f1bc3b-6c56-4146-b73e-0cf5dda18116","Type":"ContainerStarted","Data":"5dda0f9bea8cb7b74283cd5f735f4c55971ca32e85a2ef68283d329225a306aa"} Oct 30 07:30:53.670716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.639578 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podStartSLOduration=1.639555211 podStartE2EDuration="1.639555211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:53.619469976 +0000 UTC m=+2017.310387417" watchObservedRunningTime="2026-10-30 07:30:53.639555211 +0000 UTC m=+2017.330472649" Oct 30 07:30:53.696663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.694969 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:53.946071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:53.945995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:54.035527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.035498 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:30:54.115789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.115759 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:30:54.117767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.117735 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:54.117916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.117799 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:54.603493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.603454 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" event={"ID":"8e2fe803-a207-49f5-90be-67c85d293167","Type":"ContainerStarted","Data":"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa"} Oct 30 07:30:54.603852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.603505 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" event={"ID":"8e2fe803-a207-49f5-90be-67c85d293167","Type":"ContainerStarted","Data":"dc7d628c645ee53c5c0395e8cea0b1c8c8dd88b500ddb1307348b75e1888c4d6"} Oct 30 07:30:54.648008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.647958 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" podStartSLOduration=1.647938801 podStartE2EDuration="1.647938801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:30:54.645050595 +0000 UTC m=+2018.335968033" watchObservedRunningTime="2026-10-30 07:30:54.647938801 +0000 UTC m=+2018.338856281" Oct 30 07:30:54.697893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:54.697854 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:30:55.116222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:55.116185 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:55.116725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:55.116241 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:55.770037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:55.770006 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:56.116911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:56.116880 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:56.117334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:56.116938 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:57.116325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:57.116291 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:57.116514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:57.116349 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:57.606807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:57.606770 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:30:58.116122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:58.116091 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:58.116302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:58.116142 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:59.116038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:59.116007 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:30:59.116473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:59.116060 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:30:59.321103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:30:59.321072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:00.116180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:00.116148 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 07:31:00.116713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:00.116203 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 07:31:00.875380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:00.875343 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:01.145991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.145896 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:31:01.146466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.146404 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:31:01.147339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.147311 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:31:01.864110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.864074 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:31:01.868972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.868945 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.885478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.885448 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"kube-root-ca.crt\"" Oct 30 07:31:01.885667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.885527 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"openshift-service-ca.crt\"" Oct 30 07:31:01.890531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.890504 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:31:01.897012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.896972 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"auth-test-model-bf563a-kserve-self-signed-certs\"" Oct 30 07:31:01.897181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.897112 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"default-dockercfg-52dcn\"" Oct 30 07:31:01.936354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.936316 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:01.960552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.960517 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.960552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.960553 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.960881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.960829 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6959\" (UniqueName: \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.960949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.960883 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.961009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.960980 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:01.961085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:01.961068 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062294 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062348 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062531 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6959\" (UniqueName: \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062576 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062602 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.062771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062658 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.063011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062768 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.063011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.062814 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.063124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.063025 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.065595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.065567 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.065889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.065869 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.086620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.086571 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6959\" (UniqueName: \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\") pod \"auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.201771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.200882 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:02.562817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.562786 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:31:02.676908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.676866 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" event={"ID":"1d48065e-fd95-4a9e-a2b7-2f300bca75bd","Type":"ContainerStarted","Data":"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718"} Oct 30 07:31:02.677091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.676920 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" event={"ID":"1d48065e-fd95-4a9e-a2b7-2f300bca75bd","Type":"ContainerStarted","Data":"f0dd645baef28fe9f2fd52dfd59b80d1d90fd419b5e083911cb1623dce574762"} Oct 30 07:31:02.796947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.796881 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" podStartSLOduration=1.796859935 podStartE2EDuration="1.796859935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:02.796361528 +0000 UTC m=+2026.487278966" watchObservedRunningTime="2026-10-30 07:31:02.796859935 +0000 UTC m=+2026.487777372" Oct 30 07:31:02.905272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:02.905150 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:03.201910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:03.201830 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:03.698290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:03.698261 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:31:03.711081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:03.711055 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:31:04.629908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:04.629871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:04.711948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:04.711911 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:31:04.834017 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:31:04.833974 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:31:06.026153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:06.026111 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:06.277235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:06.277104 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:07.870958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.870919 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:31:07.877407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.877380 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.898849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.898813 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:31:07.906990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.906960 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4b0e81\"/\"openshift-service-ca.crt\"" Oct 30 07:31:07.906990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.906958 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4b0e81\"/\"default-dockercfg-5pq6d\"" Oct 30 07:31:07.906990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.906973 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-4b0e81\"/\"kube-root-ca.crt\"" Oct 30 07:31:07.907293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.907069 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-4b0e81\"/\"auth-test-model-4b0e81-kserve-self-signed-certs\"" Oct 30 07:31:07.925663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.924354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:07.961413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961379 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.961594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961453 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.961594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961533 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.961779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961675 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xxqb\" (UniqueName: \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.961919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961895 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:07.962084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:07.961938 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.033105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.033076 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc"] Oct 30 07:31:08.036591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.036561 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0893d6e3-855sc"] Oct 30 07:31:08.051319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.051285 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:08.063879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.063842 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.063899 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.063996 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064027 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064063 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064147 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xxqb\" (UniqueName: \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064338 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064615 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.064774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.064666 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.066462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.066437 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.066873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.066851 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.088235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.088198 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xxqb\" (UniqueName: \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\") pod \"auth-test-model-4b0e81-kserve-848b94fb45-m2jjh\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.193647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.193473 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:08.431930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.431899 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e45d8d4-4662-4bbd-b218-a4fb338f2b23" path="/var/lib/kubelet/pods/0e45d8d4-4662-4bbd-b218-a4fb338f2b23/volumes" Oct 30 07:31:08.524456 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:31:08.524423 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74dbc724_84a7_4263_a1f2_8224143fe224.slice/crio-d68499ceab564b9dea62ccaf9d2c63007bb05155dab4bfdd63c4a73653ab38a8 WatchSource:0}: Error finding container d68499ceab564b9dea62ccaf9d2c63007bb05155dab4bfdd63c4a73653ab38a8: Status 404 returned error can't find the container with id d68499ceab564b9dea62ccaf9d2c63007bb05155dab4bfdd63c4a73653ab38a8 Oct 30 07:31:08.526068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.525967 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:31:08.750155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.750115 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" event={"ID":"74dbc724-84a7-4263-a1f2-8224143fe224","Type":"ContainerStarted","Data":"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9"} Oct 30 07:31:08.750155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.750160 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" event={"ID":"74dbc724-84a7-4263-a1f2-8224143fe224","Type":"ContainerStarted","Data":"d68499ceab564b9dea62ccaf9d2c63007bb05155dab4bfdd63c4a73653ab38a8"} Oct 30 07:31:08.793282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:08.793234 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" podStartSLOduration=1.793219776 podStartE2EDuration="1.793219776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:08.791844154 +0000 UTC m=+2032.482761593" watchObservedRunningTime="2026-10-30 07:31:08.793219776 +0000 UTC m=+2032.484137213" Oct 30 07:31:09.194475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:09.194398 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:09.717261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:09.717228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:11.315209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:11.315173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:12.202183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:12.202147 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:12.215317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:12.215275 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:12.806909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:12.806881 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:31:13.003298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:13.003248 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:14.649879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:14.649842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:16.031216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:16.031177 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl"] Oct 30 07:31:16.043258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:16.043230 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-837f9426-ghhnl"] Oct 30 07:31:16.343128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:16.343038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:16.433814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:16.433785 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4dff8e-8550-409c-a7da-e50eb01a55ef" path="/var/lib/kubelet/pods/8d4dff8e-8550-409c-a7da-e50eb01a55ef/volumes" Oct 30 07:31:16.755156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:16.755072 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:31:18.090655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.089821 5894 scope.go:122] "RemoveContainer" containerID="f3c80d7c52c8c030dec1da2e2495b540aee62e8a879a0dd223aca49361a9dc00" Oct 30 07:31:18.110972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.110941 5894 scope.go:122] "RemoveContainer" containerID="763c91d27a179ab56993b0d6240715532d6350fde7b1a5eed57fc6b036c23891" Oct 30 07:31:18.126272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.126226 5894 scope.go:122] "RemoveContainer" containerID="89145c483aa316fe6d81283cf5b31df57aaa853befaccac7152846f07a775fe4" Oct 30 07:31:18.152424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.152391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:18.206018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.197613 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:18.224665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.218508 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:18.855386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:18.855357 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:31:19.957700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:19.957670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:20.222799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:20.222698 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:20.704053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:20.704022 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:21.219690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:21.219655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:21.385288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:21.369016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:21.720596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:21.720560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:22.031267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:22.031235 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk"] Oct 30 07:31:22.043285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:22.043253 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9cf88-78zjk"] Oct 30 07:31:22.434173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:22.434087 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb7152b-25cd-436c-a0a1-9ac2f0afe187" path="/var/lib/kubelet/pods/fbb7152b-25cd-436c-a0a1-9ac2f0afe187/volumes" Oct 30 07:31:23.307219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:23.307177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:25.032071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:25.032036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:25.164474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:25.164432 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:27.018563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.018531 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:27.475566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.475531 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:27.596991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.596924 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:31:27.598571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.598546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:27.601954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.601930 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.621139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.621114 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:31:27.629100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.629071 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-46dac2\"/\"openshift-service-ca.crt\"" Oct 30 07:31:27.629260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.629119 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-46dac2\"/\"kube-root-ca.crt\"" Oct 30 07:31:27.629260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.629176 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-46dac2\"/\"autores-model-46dac2-kserve-self-signed-certs\"" Oct 30 07:31:27.629805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.629792 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-46dac2\"/\"default-dockercfg-5twqv\"" Oct 30 07:31:27.675686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.675616 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.675937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.675886 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.676173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.676152 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.676273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.676218 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.676338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.676302 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.676447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.676403 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjgk5\" (UniqueName: \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.719786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.719746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:27.778299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.778507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778343 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.778507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778376 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.778507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778428 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjgk5\" (UniqueName: \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.778940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778578 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.778940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.778625 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.779681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.779240 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.780022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.779990 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.780266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.780247 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.783328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.783297 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.783940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.783920 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.805208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.805152 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjgk5\" (UniqueName: \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\") pod \"autores-model-46dac2-kserve-547f5c48df-p5glg\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:27.950655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:27.949873 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:28.317183 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:31:28.317118 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87d35089_949a_41e6_ab91_b629b70507f9.slice/crio-0d55782eb3d464603d9a05e737a9d49bb70187a5e940d3cb2ce44d0b2cbbe0fd WatchSource:0}: Error finding container 0d55782eb3d464603d9a05e737a9d49bb70187a5e940d3cb2ce44d0b2cbbe0fd: Status 404 returned error can't find the container with id 0d55782eb3d464603d9a05e737a9d49bb70187a5e940d3cb2ce44d0b2cbbe0fd Oct 30 07:31:28.319177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:28.319151 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:31:28.685413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:28.685279 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:28.828299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:28.828264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:28.958954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:28.958870 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" event={"ID":"87d35089-949a-41e6-ab91-b629b70507f9","Type":"ContainerStarted","Data":"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310"} Oct 30 07:31:28.958954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:28.958918 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" event={"ID":"87d35089-949a-41e6-ab91-b629b70507f9","Type":"ContainerStarted","Data":"0d55782eb3d464603d9a05e737a9d49bb70187a5e940d3cb2ce44d0b2cbbe0fd"} Oct 30 07:31:29.004654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:29.004587 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" podStartSLOduration=2.004572223 podStartE2EDuration="2.004572223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:29.000667374 +0000 UTC m=+2052.691584836" watchObservedRunningTime="2026-10-30 07:31:29.004572223 +0000 UTC m=+2052.695489723" Oct 30 07:31:29.952782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:29.952755 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:30.029835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:30.029798 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6"] Oct 30 07:31:30.042375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:30.042346 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf1eb026-fhfs6"] Oct 30 07:31:30.432496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:30.432460 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67765ca0-2d32-43ad-a0ae-cab37e23dffc" path="/var/lib/kubelet/pods/67765ca0-2d32-43ad-a0ae-cab37e23dffc/volumes" Oct 30 07:31:30.614237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:30.614208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:32.236933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:32.236899 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:34.035203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:34.035168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:34.830327 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:31:34.830291 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:31:35.041046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:35.041013 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr"] Oct 30 07:31:35.052506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:35.052478 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-240cf2df-xnbpr"] Oct 30 07:31:35.908316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:35.907916 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:36.432255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.432212 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95e69efc-3c70-4a68-987e-4f8112f3b097" path="/var/lib/kubelet/pods/95e69efc-3c70-4a68-987e-4f8112f3b097/volumes" Oct 30 07:31:36.809089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.809049 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:31:36.831015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.830970 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.837482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.837451 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:31:36.845985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.845948 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.846165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.846034 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.846165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.846088 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsnz\" (UniqueName: \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.846302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.846228 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.846302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.846259 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.846419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.846343 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.847232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.847206 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"openshift-service-ca.crt\"" Oct 30 07:31:36.859655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.859617 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"kube-root-ca.crt\"" Oct 30 07:31:36.859813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.859617 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"default-dockercfg-6rzbr\"" Oct 30 07:31:36.859813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.859710 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bd38cf\"/\"auth-test-model-bd38cf-kserve-self-signed-certs\"" Oct 30 07:31:36.864524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.864493 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:36.947463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsnz\" (UniqueName: \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.947709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947522 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.947709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947560 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.947709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947671 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.947897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947715 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.947897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.947770 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.948138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.948108 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.948311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.948287 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.948311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.948127 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.950178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.950152 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.950852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.950829 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:36.981256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:36.981217 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsnz\" (UniqueName: \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\") pod \"auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:37.149262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:37.149167 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:37.482718 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:31:37.482686 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7ff55f4_3b03_42f2_a21d_0f0c0221ea96.slice/crio-2a7b98d7556c81b680ddc7fa8e3a0de6acf0e87901349814bf8ac12fe77f5ebc WatchSource:0}: Error finding container 2a7b98d7556c81b680ddc7fa8e3a0de6acf0e87901349814bf8ac12fe77f5ebc: Status 404 returned error can't find the container with id 2a7b98d7556c81b680ddc7fa8e3a0de6acf0e87901349814bf8ac12fe77f5ebc Oct 30 07:31:37.485695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:37.485668 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:31:37.728095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:37.728062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:37.953320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:37.953287 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:37.967147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:37.967113 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:38.043166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:38.043123 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" event={"ID":"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96","Type":"ContainerStarted","Data":"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5"} Oct 30 07:31:38.043352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:38.043174 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" event={"ID":"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96","Type":"ContainerStarted","Data":"2a7b98d7556c81b680ddc7fa8e3a0de6acf0e87901349814bf8ac12fe77f5ebc"} Oct 30 07:31:38.054536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:38.054490 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:31:38.086629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:38.086565 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" podStartSLOduration=2.086551927 podStartE2EDuration="2.086551927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:31:38.084317125 +0000 UTC m=+2061.775234561" watchObservedRunningTime="2026-10-30 07:31:38.086551927 +0000 UTC m=+2061.777469363" Oct 30 07:31:38.149714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:38.149621 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:39.420892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:39.420849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:41.004386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:41.004351 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:42.043395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:42.043362 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls"] Oct 30 07:31:42.054695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:42.054662 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8774e7c6-d4cls"] Oct 30 07:31:42.431789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:42.431704 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f53cf7d-e409-45ce-89a4-ff11ff2c4568" path="/var/lib/kubelet/pods/8f53cf7d-e409-45ce-89a4-ff11ff2c4568/volumes" Oct 30 07:31:42.731167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:42.731083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:44.434543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:44.434509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:46.379659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:46.377998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:47.150079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:47.150047 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:47.162963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:47.162938 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:48.038797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:48.038759 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd"] Oct 30 07:31:48.051492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:48.051454 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96064fb5-r6wdd"] Oct 30 07:31:48.128672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:48.128603 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:48.186601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:48.186563 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:31:48.432210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:48.432124 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c3051c-c378-4b76-841e-124b2c88977a" path="/var/lib/kubelet/pods/82c3051c-c378-4b76-841e-124b2c88977a/volumes" Oct 30 07:31:49.706585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:49.706546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:49.828964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:49.828918 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:49.943183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:49.943146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:51.553435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:51.553394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:53.029673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:53.029623 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj"] Oct 30 07:31:53.041716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:53.041686 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4cb83fbe-tc4dj"] Oct 30 07:31:53.234160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:53.234126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:54.432470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:54.432433 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fe64f4f-1a8f-459b-937c-745dd582d680" path="/var/lib/kubelet/pods/3fe64f4f-1a8f-459b-937c-745dd582d680/volumes" Oct 30 07:31:54.829866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:54.829836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:56.515240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:56.515207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:58.210569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:58.210531 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:59.031066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:59.031034 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k"] Oct 30 07:31:59.043981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:59.043943 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fb3ce34-sw24k"] Oct 30 07:31:59.817277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:59.817239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:31:59.892031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:59.891947 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:31:59.947743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:31:59.947596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:00.210936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.210856 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z"] Oct 30 07:32:00.216721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.216695 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.233864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.233831 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:32:00.236967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.236927 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z"] Oct 30 07:32:00.255180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.254903 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:32:00.306907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.306880 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:32:00.306907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.306879 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:32:00.342554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.342515 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.342554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.342556 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p84mn\" (UniqueName: \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.342844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.342802 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.432220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.432185 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c2632c-1988-45c7-9240-1dfdde0ddda5" path="/var/lib/kubelet/pods/a5c2632c-1988-45c7-9240-1dfdde0ddda5/volumes" Oct 30 07:32:00.443772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.443740 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.443935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.443885 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.443935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.443908 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p84mn\" (UniqueName: \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.444590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.444572 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.446252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.446222 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.467946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.467861 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p84mn\" (UniqueName: \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\") pod \"maas-api-revoke-keys-e2e-autores-46dac2-mxm4z\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.538487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.538446 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:00.854884 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:32:00.854851 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d8c624_2033_454f_a3c3_06bfa1d4b0ab.slice/crio-571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af WatchSource:0}: Error finding container 571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af: Status 404 returned error can't find the container with id 571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af Oct 30 07:32:00.856747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:00.856725 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z"] Oct 30 07:32:01.332025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:01.331974 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" event={"ID":"23d8c624-2033-454f-a3c3-06bfa1d4b0ab","Type":"ContainerStarted","Data":"ed3fce97108153dc46cba4921de67d28f9fdf43a1f6b860b992ba73e097f533a"} Oct 30 07:32:01.332221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:01.332082 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" event={"ID":"23d8c624-2033-454f-a3c3-06bfa1d4b0ab","Type":"ContainerStarted","Data":"571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af"} Oct 30 07:32:01.377651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:01.377576 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" podStartSLOduration=1.377560276 podStartE2EDuration="1.377560276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:01.376376496 +0000 UTC m=+2085.067293934" watchObservedRunningTime="2026-10-30 07:32:01.377560276 +0000 UTC m=+2085.068477714" Oct 30 07:32:01.630753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:01.630716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:02.342052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:02.342014 5894 generic.go:356] "Generic (PLEG): container finished" podID="23d8c624-2033-454f-a3c3-06bfa1d4b0ab" containerID="ed3fce97108153dc46cba4921de67d28f9fdf43a1f6b860b992ba73e097f533a" exitCode=0 Oct 30 07:32:02.342525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:02.342098 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" event={"ID":"23d8c624-2033-454f-a3c3-06bfa1d4b0ab","Type":"ContainerDied","Data":"ed3fce97108153dc46cba4921de67d28f9fdf43a1f6b860b992ba73e097f533a"} Oct 30 07:32:03.217779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.217746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:03.511109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.511075 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:03.590746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.590716 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\" (UniqueName: \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\") pod \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " Oct 30 07:32:03.590930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.590862 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\") pod \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " Oct 30 07:32:03.591053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.591031 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\") pod \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\" (UID: \"23d8c624-2033-454f-a3c3-06bfa1d4b0ab\") " Oct 30 07:32:03.591227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.591189 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca" pod "23d8c624-2033-454f-a3c3-06bfa1d4b0ab" (UID: "23d8c624-2033-454f-a3c3-06bfa1d4b0ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:32:03.591575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.591558 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:03.592989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.592960 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn" pod "23d8c624-2033-454f-a3c3-06bfa1d4b0ab" (UID: "23d8c624-2033-454f-a3c3-06bfa1d4b0ab"). InnerVolumeSpecName "kube-api-access-p84mn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:03.593107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.592994 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca" pod "23d8c624-2033-454f-a3c3-06bfa1d4b0ab" (UID: "23d8c624-2033-454f-a3c3-06bfa1d4b0ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:32:03.692778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.692744 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:03.692778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:03.692772 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p84mn\" (UniqueName: \"kubernetes.io/projected/23d8c624-2033-454f-a3c3-06bfa1d4b0ab-kube-api-access-p84mn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:04.359629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:04.359598 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" Oct 30 07:32:04.359814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:04.359626 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z" event={"ID":"23d8c624-2033-454f-a3c3-06bfa1d4b0ab","Type":"ContainerDied","Data":"571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af"} Oct 30 07:32:04.359814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:04.359699 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571f2b2814d39eb33d32cbdcffe2986fd7c86b9b87c5f09591e04520792e60af" Oct 30 07:32:04.833355 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:32:04.833319 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:32:04.951382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:04.951349 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:06.525121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:06.525082 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:06.754660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:06.753210 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:32:08.490839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:08.490798 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:09.031094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:09.031057 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688"] Oct 30 07:32:09.043414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:09.043372 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9444437c-5w688"] Oct 30 07:32:10.108956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:10.108915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:10.431591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:10.431506 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f14311-27f1-48c5-8e78-6bc3cd9f3caf" path="/var/lib/kubelet/pods/40f14311-27f1-48c5-8e78-6bc3cd9f3caf/volumes" Oct 30 07:32:10.836782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:10.836740 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:10.948228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:10.948191 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:11.824026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:11.823994 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:12.030334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:12.030303 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x"] Oct 30 07:32:12.042731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:12.042701 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee30481-pfk8x"] Oct 30 07:32:12.433889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:12.433853 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4a2911a-62aa-4753-a002-156ef9013c14" path="/var/lib/kubelet/pods/a4a2911a-62aa-4753-a002-156ef9013c14/volumes" Oct 30 07:32:13.505846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:13.505812 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:14.943524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:14.943479 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" podUID="87d35089-949a-41e6-ab91-b629b70507f9" containerName="main" containerID="cri-o://ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310" gracePeriod=45 Oct 30 07:32:15.044187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.044146 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" podUID="87d35089-949a-41e6-ab91-b629b70507f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.196:8000/ready\": dial tcp 10.134.0.196:8000: connect: connection refused" Oct 30 07:32:15.214747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.214713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:15.220880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.220857 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271236 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271420 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\" (UniqueName: \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271492 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271528 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.271665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.271629 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\" (UniqueName: \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\") pod \"87d35089-949a-41e6-ab91-b629b70507f9\" (UID: \"87d35089-949a-41e6-ab91-b629b70507f9\") " Oct 30 07:32:15.279242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.279140 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:15.287890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.279470 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:15.287890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.279783 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:15.288371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.288258 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:32:15.288371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.288329 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:32:15.288713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.288685 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5" pod "87d35089-949a-41e6-ab91-b629b70507f9" (UID: "87d35089-949a-41e6-ab91-b629b70507f9"). InnerVolumeSpecName "kube-api-access-bjgk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:32:15.373715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373680 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87d35089-949a-41e6-ab91-b629b70507f9-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.373715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373709 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.373971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373738 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.373971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373747 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjgk5\" (UniqueName: \"kubernetes.io/projected/87d35089-949a-41e6-ab91-b629b70507f9-kube-api-access-bjgk5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.373971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373757 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.373971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.373765 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87d35089-949a-41e6-ab91-b629b70507f9-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:32:15.476012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.475934 5894 generic.go:356] "Generic (PLEG): container finished" podID="87d35089-949a-41e6-ab91-b629b70507f9" containerID="ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310" exitCode=0 Oct 30 07:32:15.476012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.475988 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" Oct 30 07:32:15.476217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.476020 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" event={"ID":"87d35089-949a-41e6-ab91-b629b70507f9","Type":"ContainerDied","Data":"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310"} Oct 30 07:32:15.476217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.476069 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg" event={"ID":"87d35089-949a-41e6-ab91-b629b70507f9","Type":"ContainerDied","Data":"0d55782eb3d464603d9a05e737a9d49bb70187a5e940d3cb2ce44d0b2cbbe0fd"} Oct 30 07:32:15.476217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.476086 5894 scope.go:122] "RemoveContainer" containerID="ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310" Oct 30 07:32:15.489392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.489369 5894 scope.go:122] "RemoveContainer" containerID="ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310" Oct 30 07:32:15.490587 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:32:15.490560 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310\": container with ID starting with ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310 not found: ID does not exist" containerID="ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310" Oct 30 07:32:15.490740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.490593 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310"} err="failed to get container status \"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310\": rpc error: code = NotFound desc = could not find container \"ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310\": container with ID starting with ab96d841ed02433734d95587edc54a76dd760970f8310f82a638b168c3b65310 not found: ID does not exist" Oct 30 07:32:15.531574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.531537 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:32:15.534985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.534964 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-46dac2/autores-model-46dac2-kserve-547f5c48df-p5glg"] Oct 30 07:32:15.988514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:15.988478 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:16.132744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:16.132711 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:16.432360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:16.432325 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d35089-949a-41e6-ab91-b629b70507f9" path="/var/lib/kubelet/pods/87d35089-949a-41e6-ab91-b629b70507f9/volumes" Oct 30 07:32:16.718660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:16.718549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:17.410869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:17.410835 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:17.546976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:17.546934 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:17.587819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:17.587783 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z"] Oct 30 07:32:17.629757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:17.629719 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-46dac2-mxm4z"] Oct 30 07:32:17.656015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:17.655980 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:18.117047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.117016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:18.253450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.253422 5894 scope.go:122] "RemoveContainer" containerID="2bc843af641086df5605d1f24aeea44745a9b212b70cd3ec57ed304055375a89" Oct 30 07:32:18.264726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.264702 5894 scope.go:122] "RemoveContainer" containerID="01e0a844f8fd45eea1b0f726fdc8c66720efc05920b9a528ca9287056d5aaa63" Oct 30 07:32:18.274977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.274949 5894 scope.go:122] "RemoveContainer" containerID="1ad62cd2a7dac4597bcbb88f0a73d3be77d7a06828bd16ccc70b8b0d2b5bf96e" Oct 30 07:32:18.287839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.287813 5894 scope.go:122] "RemoveContainer" containerID="4fc31b8ba67f8aa8a973015c68bf95e51812e53c7c859c29921835271bfcfb4f" Oct 30 07:32:18.297751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.297730 5894 scope.go:122] "RemoveContainer" containerID="ddc41a192343f6950162bff76c552577dee889a8eb536d3243120aeb489a8ca2" Oct 30 07:32:18.433407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:18.433316 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d8c624-2033-454f-a3c3-06bfa1d4b0ab" path="/var/lib/kubelet/pods/23d8c624-2033-454f-a3c3-06bfa1d4b0ab/volumes" Oct 30 07:32:19.294908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:19.294874 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:20.938900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:20.938868 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:21.156790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:21.156750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:22.660027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:22.659991 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:24.396137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.396100 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:24.526413 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.526378 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:32:24.527103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527082 5894 cpu_manager.go:414] "containerMap: removing container" podUID="87d35089-949a-41e6-ab91-b629b70507f9" containerName="main" Oct 30 07:32:24.527103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527103 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d35089-949a-41e6-ab91-b629b70507f9" containerName="main" Oct 30 07:32:24.527241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527126 5894 cpu_manager.go:414] "containerMap: removing container" podUID="23d8c624-2033-454f-a3c3-06bfa1d4b0ab" containerName="revoke-keys" Oct 30 07:32:24.527241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527132 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d8c624-2033-454f-a3c3-06bfa1d4b0ab" containerName="revoke-keys" Oct 30 07:32:24.527241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527223 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d35089-949a-41e6-ab91-b629b70507f9" containerName="main" Oct 30 07:32:24.527241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.527237 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d8c624-2033-454f-a3c3-06bfa1d4b0ab" containerName="revoke-keys" Oct 30 07:32:24.533129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.533103 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.553138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.553104 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:32:24.562683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.560540 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"default-dockercfg-zlnwx\"" Oct 30 07:32:24.562683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.560560 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"openshift-service-ca.crt\"" Oct 30 07:32:24.562683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.560552 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"kube-root-ca.crt\"" Oct 30 07:32:24.562683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.560658 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"auth-test-model-d89593-kserve-self-signed-certs\"" Oct 30 07:32:24.590296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.590264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:24.625208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625175 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6qbh\" (UniqueName: \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.625434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625320 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.625434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625349 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.625573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625437 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.625573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625505 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.625724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.625578 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727208 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727344 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727473 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6qbh\" (UniqueName: \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727602 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727627 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727683 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.727808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.727683 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.728050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.728025 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.729978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.729956 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.730432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.730404 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.743564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.743535 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:24.751106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.751077 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6qbh\" (UniqueName: \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\") pod \"auth-test-model-d89593-kserve-9fd8d4bfc-phs2q\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.855659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.854880 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:24.880367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:24.880336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:25.100966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.100930 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:25.329839 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:32:25.329810 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bc7b89e_44be_40ed_9258_a1651cdbf0ca.slice/crio-94e0aecc5888fe80cf2b004786f26a0ecaa5da99fc9389e0e1dbcc99eb737fff WatchSource:0}: Error finding container 94e0aecc5888fe80cf2b004786f26a0ecaa5da99fc9389e0e1dbcc99eb737fff: Status 404 returned error can't find the container with id 94e0aecc5888fe80cf2b004786f26a0ecaa5da99fc9389e0e1dbcc99eb737fff Oct 30 07:32:25.330417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.330380 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:32:25.567530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.567495 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" event={"ID":"6bc7b89e-44be-40ed-9258-a1651cdbf0ca","Type":"ContainerStarted","Data":"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844"} Oct 30 07:32:25.567933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.567546 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" event={"ID":"6bc7b89e-44be-40ed-9258-a1651cdbf0ca","Type":"ContainerStarted","Data":"94e0aecc5888fe80cf2b004786f26a0ecaa5da99fc9389e0e1dbcc99eb737fff"} Oct 30 07:32:25.611921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.611801 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" podStartSLOduration=1.611782016 podStartE2EDuration="1.611782016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:25.608875323 +0000 UTC m=+2109.299792760" watchObservedRunningTime="2026-10-30 07:32:25.611782016 +0000 UTC m=+2109.302699454" Oct 30 07:32:25.862372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.862269 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:25.901690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:25.901655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:26.031194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.031164 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4"] Oct 30 07:32:26.034009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.033987 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3debd49f-72cz4"] Oct 30 07:32:26.135852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.135768 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:26.293272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.293232 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:26.430733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.429877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:26.433562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:26.433526 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f8dded-a31f-4e52-82d4-b0b636ecb8f3" path="/var/lib/kubelet/pods/40f8dded-a31f-4e52-82d4-b0b636ecb8f3/volumes" Oct 30 07:32:27.783679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:27.783629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:28.831111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:28.831078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:29.479890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:29.479848 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:29.721301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:29.721259 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:29.838943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:29.838907 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:31.032252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:31.032223 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc"] Oct 30 07:32:31.044210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:31.044179 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a1d269-njkmc"] Oct 30 07:32:31.325972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:31.325863 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:31.442948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:31.442916 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:31.563622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:31.563582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:32.431730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:32.431691 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676c5fd6-e168-4948-ab8f-8d21b9c69b25" path="/var/lib/kubelet/pods/676c5fd6-e168-4948-ab8f-8d21b9c69b25/volumes" Oct 30 07:32:33.113935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:33.113890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:34.827552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:34.827514 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:34.829617 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:32:34.829582 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:32:34.855882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:34.855853 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:34.887122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:34.887090 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:35.670173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:35.670142 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:32:36.418521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:36.418490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:38.051297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:38.051255 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:39.693860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:39.693830 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:41.424252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:41.424214 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:32:41.433857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:41.433826 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:43.256586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:43.256553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:44.961314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:44.961279 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:46.733205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:46.733171 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:48.034781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:48.034736 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc"] Oct 30 07:32:48.044971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:48.044937 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c947a35-z7gnc"] Oct 30 07:32:48.423542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:48.423449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:48.431495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:48.431465 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bd209d-3179-49a9-bf78-204d737783cd" path="/var/lib/kubelet/pods/43bd209d-3179-49a9-bf78-204d737783cd/volumes" Oct 30 07:32:48.970784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:48.970746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:50.011457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:50.011420 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:51.642655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:51.642601 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:53.273626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:53.273589 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:53.403336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:53.403298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:53.526130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:53.526043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:55.031901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:55.031869 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:56.210120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.210089 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:32:56.214724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.214693 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.235979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.235946 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"default-dockercfg-76wzq\"" Oct 30 07:32:56.249145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.249114 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"openshift-service-ca.crt\"" Oct 30 07:32:56.249344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.249246 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"kube-root-ca.crt\"" Oct 30 07:32:56.254457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.254429 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:32:56.277914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.277834 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:56.317110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317077 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.317319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317118 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.317319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317287 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.317444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317379 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlqjj\" (UniqueName: \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.317444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317418 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.317559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.317476 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.324726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.324698 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"auth-test-model-054ef1-kserve-self-signed-certs\"" Oct 30 07:32:56.418874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.418835 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.418874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.418877 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419044 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419135 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlqjj\" (UniqueName: \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419173 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419239 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419343 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419372 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.419893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.419860 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.421429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.421398 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.421578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.421561 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.466533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.461308 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlqjj\" (UniqueName: \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\") pod \"auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.536691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.535894 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:32:56.836843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.836810 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:56.902454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:56.902414 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:32:57.828834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:57.828797 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" event={"ID":"4793fe07-dee8-45e2-bb0f-f8520908bef9","Type":"ContainerStarted","Data":"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361"} Oct 30 07:32:57.828834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:57.828839 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" event={"ID":"4793fe07-dee8-45e2-bb0f-f8520908bef9","Type":"ContainerStarted","Data":"af916426b9377f755005a16d6dc9750058fb68a922e7574a35ff20f8a3d92750"} Oct 30 07:32:57.873353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:57.873294 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" podStartSLOduration=1.873279283 podStartE2EDuration="1.873279283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:32:57.87143775 +0000 UTC m=+2141.562355217" watchObservedRunningTime="2026-10-30 07:32:57.873279283 +0000 UTC m=+2141.564196719" Oct 30 07:32:58.523053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:58.523016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:32:58.536776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:32:58.536742 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:33:00.008507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:00.008472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:00.235778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:00.235741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:01.840930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:01.840897 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:03.031903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:03.031868 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4"] Oct 30 07:33:03.044097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:03.044065 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e7fb5e-qrbj4"] Oct 30 07:33:03.433510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:03.433423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:04.031516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:04.031488 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th"] Oct 30 07:33:04.043597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:04.043571 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64f12108-k92th"] Oct 30 07:33:04.432393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:04.432304 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b243156-786c-4491-90ff-53d793280097" path="/var/lib/kubelet/pods/4b243156-786c-4491-90ff-53d793280097/volumes" Oct 30 07:33:04.432819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:04.432803 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35f67f3-68c9-4ad9-b8d3-170deb65e034" path="/var/lib/kubelet/pods/d35f67f3-68c9-4ad9-b8d3-170deb65e034/volumes" Oct 30 07:33:04.833935 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:33:04.833892 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:33:05.156535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:05.156449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:06.538048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:06.538008 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:33:06.550417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:06.550386 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:33:06.706073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:06.706040 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:06.917159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:06.917074 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:33:08.422493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:08.422456 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:10.134303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:10.134268 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:11.032193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:11.032159 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g"] Oct 30 07:33:11.044663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:11.044613 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c6bda3c-zc24g"] Oct 30 07:33:11.818032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:11.817997 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:12.433011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:12.432980 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef3436c9-8dfc-4cab-8b98-086711d6484b" path="/var/lib/kubelet/pods/ef3436c9-8dfc-4cab-8b98-086711d6484b/volumes" Oct 30 07:33:13.592233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:13.592189 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:15.377428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:15.377387 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:17.032398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:17.032351 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:18.034130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.034088 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z"] Oct 30 07:33:18.046562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.046533 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5"] Oct 30 07:33:18.049174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.049143 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4309d09-fh75z"] Oct 30 07:33:18.058862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.058837 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51dcef3-7r5j5"] Oct 30 07:33:18.359092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.359012 5894 scope.go:122] "RemoveContainer" containerID="f172b5b87b932a1b893e26b94fb880367872ec1a6bedce43b2991f74ca4edaa3" Oct 30 07:33:18.369385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.369362 5894 scope.go:122] "RemoveContainer" containerID="a47923d165e0c2fde02c690f972f69e2357fba9866d22996dbaead2cd35f18cc" Oct 30 07:33:18.378990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.378967 5894 scope.go:122] "RemoveContainer" containerID="af247ef5b449f4528d8be8c93208e0b170b25d443e65ef3cbaf17d5773c1311a" Oct 30 07:33:18.389428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.389406 5894 scope.go:122] "RemoveContainer" containerID="b1b863a1af42f1994b1bd6e9ecbcd6c7673a9bd2a5db8d479b8cbfbc97aee24a" Oct 30 07:33:18.398492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.398474 5894 scope.go:122] "RemoveContainer" containerID="75ae9bd6ff5166457de75f6415b00832376a2a1c4c83fac487da56563362c722" Oct 30 07:33:18.407395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.407372 5894 scope.go:122] "RemoveContainer" containerID="1585b220ac41bcbd0c1993e11d3891a2e1b0d9db122b5728aaf261d94db3cef5" Oct 30 07:33:18.418533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.418499 5894 scope.go:122] "RemoveContainer" containerID="2354ce7758166dc64741c57f6f60eab2db6720f419ae0d1f2554a9386fdaedd3" Oct 30 07:33:18.427483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.427468 5894 scope.go:122] "RemoveContainer" containerID="cec2cb4a2951be14722ab15926289cd6746ead542ab08dcb84ac3f89607d3f7b" Oct 30 07:33:18.432919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.432889 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fc17ef-e735-40f1-8325-064ddacb3719" path="/var/lib/kubelet/pods/10fc17ef-e735-40f1-8325-064ddacb3719/volumes" Oct 30 07:33:18.433451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.433423 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f" path="/var/lib/kubelet/pods/3640ee90-6dd7-4bda-af5e-eb4eb6e0a66f/volumes" Oct 30 07:33:18.437882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.437860 5894 scope.go:122] "RemoveContainer" containerID="2f89a2812faf317b8ddfb127fbd8bc84732d156bc1cc2a6b2dd2db80685f41f8" Oct 30 07:33:18.447353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.447333 5894 scope.go:122] "RemoveContainer" containerID="46a48f26267a91942fd7e65577d175ee6b6cee239187187179a2c6b72f55957d" Oct 30 07:33:18.463872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.463848 5894 scope.go:122] "RemoveContainer" containerID="5ce48126d217a0eafa604f6343fec0ff7d3edaae51e5b625880b967d652da8b2" Oct 30 07:33:18.765447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:18.765366 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:20.562068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:20.562035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:22.330303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:22.330267 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:23.031810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:23.031774 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8"] Oct 30 07:33:23.043867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:23.043836 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-925c0e98-p4ln8"] Oct 30 07:33:24.021369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:24.021328 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:24.431857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:24.431829 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3" path="/var/lib/kubelet/pods/80457bf1-3bfd-45ee-828a-fe6dd1d7a8d3/volumes" Oct 30 07:33:25.031557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:25.031520 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6"] Oct 30 07:33:25.043913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:25.043858 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fec99a-qclm6"] Oct 30 07:33:25.635103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:25.635071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:25.821801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:25.821769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:26.437410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:26.437366 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="636c5d5e-6e95-45ed-8fd0-d303fc792a8a" path="/var/lib/kubelet/pods/636c5d5e-6e95-45ed-8fd0-d303fc792a8a/volumes" Oct 30 07:33:27.437751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:27.437719 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:27.922827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:27.922783 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:29.073082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:29.073034 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:29.196915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:29.196880 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:30.967019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:30.966982 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:32.622781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:32.622744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:33.041450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:33.041414 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89"] Oct 30 07:33:33.053761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:33.053732 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27898f8-8hr89"] Oct 30 07:33:34.218503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:34.218465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:34.432470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:34.432438 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872c617e-e333-4f1d-884d-e7fd53e38928" path="/var/lib/kubelet/pods/872c617e-e333-4f1d-884d-e7fd53e38928/volumes" Oct 30 07:33:34.839091 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:33:34.839045 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:33:35.939171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:35.939134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:37.596622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:37.596587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:39.125053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:39.125014 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:40.738217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:40.738178 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:42.031735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:42.031700 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw"] Oct 30 07:33:42.044155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:42.044121 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91472310-pjkgw"] Oct 30 07:33:42.432029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:42.431949 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e059f2c-6409-40cd-ac90-1fa5b247482b" path="/var/lib/kubelet/pods/0e059f2c-6409-40cd-ac90-1fa5b247482b/volumes" Oct 30 07:33:42.541201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:42.541160 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:44.329528 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:44.329496 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:33:44.410893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:44.410832 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:46.258015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:46.257976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:47.914294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:47.914258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:49.035297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:49.035260 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj"] Oct 30 07:33:49.048067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:49.048039 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac73e1d-mb2mj"] Oct 30 07:33:49.630033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:49.630000 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:50.020386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:50.020292 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:50.432309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:50.432275 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a166e2fa-2d80-4533-947a-fc83128a4d6f" path="/var/lib/kubelet/pods/a166e2fa-2d80-4533-947a-fc83128a4d6f/volumes" Oct 30 07:33:51.350049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:51.350011 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:52.936482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:52.936446 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:53.594043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.593995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:53.868962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.868885 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62"] Oct 30 07:33:53.876609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.876581 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:53.897257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.897206 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62"] Oct 30 07:33:53.902908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.902876 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:33:53.903817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.903792 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:33:53.958041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.958006 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:33:53.958445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.958058 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:33:53.967125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.967094 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:53.967280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.967145 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:53.967323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:53.967299 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjlpb\" (UniqueName: \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.068687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.068587 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.068687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.068652 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.068971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.068749 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjlpb\" (UniqueName: \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.069343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.069316 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.071899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.071867 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.092669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.092601 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjlpb\" (UniqueName: \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\") pod \"maas-api-revoke-keys-e2e-explicit-4fff82-bct62\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.202067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.201975 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:54.545575 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:33:54.545544 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3897eee3_282f_4273_b462_ade1907ddc22.slice/crio-583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af WatchSource:0}: Error finding container 583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af: Status 404 returned error can't find the container with id 583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af Oct 30 07:33:54.546390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.546363 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62"] Oct 30 07:33:54.710309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:54.710269 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:55.380236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:55.380206 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" event={"ID":"3897eee3-282f-4273-b462-ade1907ddc22","Type":"ContainerStarted","Data":"f606c5c94eca13a86721b4e36a3154365ae582a2d7e5b3e05bc82364497fee5e"} Oct 30 07:33:55.380774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:55.380261 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" event={"ID":"3897eee3-282f-4273-b462-ade1907ddc22","Type":"ContainerStarted","Data":"583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af"} Oct 30 07:33:55.423206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:55.423155 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" podStartSLOduration=2.423138709 podStartE2EDuration="2.423138709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:33:55.421083655 +0000 UTC m=+2199.112001089" watchObservedRunningTime="2026-10-30 07:33:55.423138709 +0000 UTC m=+2199.114056146" Oct 30 07:33:56.389893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:56.389858 5894 generic.go:356] "Generic (PLEG): container finished" podID="3897eee3-282f-4273-b462-ade1907ddc22" containerID="f606c5c94eca13a86721b4e36a3154365ae582a2d7e5b3e05bc82364497fee5e" exitCode=0 Oct 30 07:33:56.390401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:56.389940 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" event={"ID":"3897eee3-282f-4273-b462-ade1907ddc22","Type":"ContainerDied","Data":"f606c5c94eca13a86721b4e36a3154365ae582a2d7e5b3e05bc82364497fee5e"} Oct 30 07:33:56.440856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:56.440821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:57.584262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.584235 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:57.620494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.620454 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\") pod \"3897eee3-282f-4273-b462-ade1907ddc22\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " Oct 30 07:33:57.620720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.620523 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\") pod \"3897eee3-282f-4273-b462-ade1907ddc22\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " Oct 30 07:33:57.620720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.620585 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\" (UniqueName: \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\") pod \"3897eee3-282f-4273-b462-ade1907ddc22\" (UID: \"3897eee3-282f-4273-b462-ade1907ddc22\") " Oct 30 07:33:57.620933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.620895 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca" pod "3897eee3-282f-4273-b462-ade1907ddc22" (UID: "3897eee3-282f-4273-b462-ade1907ddc22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:33:57.621215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.621200 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:33:57.622791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.622770 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca" pod "3897eee3-282f-4273-b462-ade1907ddc22" (UID: "3897eee3-282f-4273-b462-ade1907ddc22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:33:57.622961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.622940 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb" pod "3897eee3-282f-4273-b462-ade1907ddc22" (UID: "3897eee3-282f-4273-b462-ade1907ddc22"). InnerVolumeSpecName "kube-api-access-hjlpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:33:57.722667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.722576 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3897eee3-282f-4273-b462-ade1907ddc22-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:33:57.722667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:57.722604 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjlpb\" (UniqueName: \"kubernetes.io/projected/3897eee3-282f-4273-b462-ade1907ddc22-kube-api-access-hjlpb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:33:58.142538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:58.142502 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:33:58.406888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:58.406809 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" Oct 30 07:33:58.406888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:58.406837 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62" event={"ID":"3897eee3-282f-4273-b462-ade1907ddc22","Type":"ContainerDied","Data":"583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af"} Oct 30 07:33:58.406888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:58.406886 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583af1c13c2ba1a6f5b71c0fef77c9df068b1ebb3e512c5dfd9aa898690e55af" Oct 30 07:33:59.831397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:33:59.831354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:01.424016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:01.423965 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:03.123544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:03.123512 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:04.621147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:04.621111 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:04.754478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:04.754447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:04.830864 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:34:04.830826 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:34:04.868506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:04.868471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:06.527610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:06.527570 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:08.230294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:08.230266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:09.656501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:09.656452 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:10.211370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.211335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:10.330919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.330890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:10.404665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.404615 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62"] Oct 30 07:34:10.417059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.417024 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4fff82-bct62"] Oct 30 07:34:10.433276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.433242 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3897eee3-282f-4273-b462-ade1907ddc22" path="/var/lib/kubelet/pods/3897eee3-282f-4273-b462-ade1907ddc22/volumes" Oct 30 07:34:10.471106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.471024 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:10.909785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:10.909752 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:12.306267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:12.306233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:13.911446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:13.911406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:15.663730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:15.663698 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:17.400817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:17.400780 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:17.561112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:17.561073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:17.673001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:17.672911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:17.911157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:17.911124 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:18.555862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.555833 5894 scope.go:122] "RemoveContainer" containerID="7eeda46e583e62d12345693a8bbdd773900d1f613ff9f2853d027432cfeea4f5" Oct 30 07:34:18.567505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.567485 5894 scope.go:122] "RemoveContainer" containerID="83b803bb089adc94ef91ab0646bda54d869b73ad92e7f1a8f45fa141e4839276" Oct 30 07:34:18.580916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.580888 5894 scope.go:122] "RemoveContainer" containerID="a493204ab1e6e8d0fd8cac9af815d7a05b2d8f4ae90fd6b9b2fd4674e88bc39d" Oct 30 07:34:18.590450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.590424 5894 scope.go:122] "RemoveContainer" containerID="cd0f0d2698359e95ddac826e9f72b729b5e45dd29e1c2306a942f499887a0c9d" Oct 30 07:34:18.602071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.602049 5894 scope.go:122] "RemoveContainer" containerID="c72de33844d28f795133c1a3040cfee37e05da9e7100c7525f667822788f6456" Oct 30 07:34:18.611755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.611728 5894 scope.go:122] "RemoveContainer" containerID="75ebecae3265cd0fe9dd084a01119d638ea3f1f5aa217fe89f141159f259404a" Oct 30 07:34:18.621052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.621033 5894 scope.go:122] "RemoveContainer" containerID="a6ebe8829c92675027680d7f6e888689cad20523408f20e31eb1a8351cd925b3" Oct 30 07:34:18.630176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.630154 5894 scope.go:122] "RemoveContainer" containerID="4a65d6e2cf507f9cbe50ce637f8f13844fd989dc8f9724670629bc6973c53053" Oct 30 07:34:18.639604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.639575 5894 scope.go:122] "RemoveContainer" containerID="705e2984306542f7b8fbde03c6164e573aaf400d64b11cee1f012a9381c9b52c" Oct 30 07:34:18.648346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:18.648316 5894 scope.go:122] "RemoveContainer" containerID="2e72a6a28d3b6eba92601b77515ed2edf0c054afc0e5c39197d2c5b2aec11924" Oct 30 07:34:19.031151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:19.031111 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:20.710519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:20.710484 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:20.988744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:20.988620 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:21.639588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:21.639548 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:21.783656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:21.782167 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:22.512625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:22.512591 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:22.863360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:22.863319 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:34:24.219903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:24.219861 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:26.140562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:26.140524 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:26.271551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:26.271515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:27.004722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.004682 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt"] Oct 30 07:34:27.005427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.005407 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3897eee3-282f-4273-b462-ade1907ddc22" containerName="revoke-keys" Oct 30 07:34:27.005495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.005429 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3897eee3-282f-4273-b462-ade1907ddc22" containerName="revoke-keys" Oct 30 07:34:27.005618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.005606 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3897eee3-282f-4273-b462-ade1907ddc22" containerName="revoke-keys" Oct 30 07:34:27.011358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.011334 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.029720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.029692 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt"] Oct 30 07:34:27.037857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.037821 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:34:27.038100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.037957 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:34:27.038100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.038019 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:34:27.039067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.039048 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:34:27.143627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.143595 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.143627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.143654 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrjp\" (UniqueName: \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.144397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.143798 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.245012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.244966 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.245205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.245027 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrjp\" (UniqueName: \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.245205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.245138 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.245935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.245897 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.247672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.247627 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.269194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.269129 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrjp\" (UniqueName: \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\") pod \"maas-api-revoke-sub-c4ded9bc-2k6bt\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.326337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.326302 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:27.473377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.473346 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc"] Oct 30 07:34:27.478803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.478781 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.497801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.497775 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc"] Oct 30 07:34:27.630727 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:34:27.630691 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5e2cc6_11c4_4960_ba19_90249a64d88e.slice/crio-a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922 WatchSource:0}: Error finding container a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922: Status 404 returned error can't find the container with id a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922 Oct 30 07:34:27.631134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.631111 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt"] Oct 30 07:34:27.641416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.641383 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" event={"ID":"ac5e2cc6-11c4-4960-ba19-90249a64d88e","Type":"ContainerStarted","Data":"a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922"} Oct 30 07:34:27.649676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.649623 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.649800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.649781 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gbp7\" (UniqueName: \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.649866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.649856 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.752240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.752202 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.752470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.752331 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gbp7\" (UniqueName: \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.752470 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.752407 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.753313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.753285 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.755082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.755057 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.777040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.777004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gbp7\" (UniqueName: \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\") pod \"maas-api-revoke-sub-afe72f5d-hrgfc\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.793501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.793471 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:27.831951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.831917 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:27.950843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:27.950812 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:28.187105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:28.187063 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc"] Oct 30 07:34:28.310089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:28.310049 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:28.651312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:28.651267 5894 generic.go:356] "Generic (PLEG): container finished" podID="ac5e2cc6-11c4-4960-ba19-90249a64d88e" containerID="6cbb779978356e556716e85fdee02cc05d02613e7c0d08f6f1755be923459495" exitCode=0 Oct 30 07:34:28.651489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:28.651343 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" event={"ID":"ac5e2cc6-11c4-4960-ba19-90249a64d88e","Type":"ContainerDied","Data":"6cbb779978356e556716e85fdee02cc05d02613e7c0d08f6f1755be923459495"} Oct 30 07:34:28.652937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:28.652905 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" event={"ID":"8692621e-332b-47dc-9b56-ba297c3a7c95","Type":"ContainerStarted","Data":"af9dc55c63de9b8da9f60c3b7a329d27f4e245c71c5f9d6f44fcd90daccdcef7"} Oct 30 07:34:29.125807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.125773 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:29.245628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.245595 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:29.661312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.661281 5894 generic.go:356] "Generic (PLEG): container finished" podID="8692621e-332b-47dc-9b56-ba297c3a7c95" containerID="a9b7cfd14787e26e0c085746bf717d47b2daa48b29394c0c0de3ba50b36a01a6" exitCode=0 Oct 30 07:34:29.661571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.661546 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" event={"ID":"8692621e-332b-47dc-9b56-ba297c3a7c95","Type":"ContainerDied","Data":"a9b7cfd14787e26e0c085746bf717d47b2daa48b29394c0c0de3ba50b36a01a6"} Oct 30 07:34:29.870402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.870376 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:29.914094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.913828 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\" (UniqueName: \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\") pod \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " Oct 30 07:34:29.914094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.913978 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\") pod \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " Oct 30 07:34:29.914380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.914353 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca" pod "ac5e2cc6-11c4-4960-ba19-90249a64d88e" (UID: "ac5e2cc6-11c4-4960-ba19-90249a64d88e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:29.914574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.914553 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\") pod \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\" (UID: \"ac5e2cc6-11c4-4960-ba19-90249a64d88e\") " Oct 30 07:34:29.915194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.915159 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:29.916428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.916399 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp" pod "ac5e2cc6-11c4-4960-ba19-90249a64d88e" (UID: "ac5e2cc6-11c4-4960-ba19-90249a64d88e"). InnerVolumeSpecName "kube-api-access-mkrjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:29.917386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:29.917351 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca" pod "ac5e2cc6-11c4-4960-ba19-90249a64d88e" (UID: "ac5e2cc6-11c4-4960-ba19-90249a64d88e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:30.016068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.016038 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac5e2cc6-11c4-4960-ba19-90249a64d88e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:30.016068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.016065 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkrjp\" (UniqueName: \"kubernetes.io/projected/ac5e2cc6-11c4-4960-ba19-90249a64d88e-kube-api-access-mkrjp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:30.140916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.140885 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:30.670704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.670660 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" Oct 30 07:34:30.670704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.670673 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt" event={"ID":"ac5e2cc6-11c4-4960-ba19-90249a64d88e","Type":"ContainerDied","Data":"a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922"} Oct 30 07:34:30.671177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.670732 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a10ba4da3439d3f8caec35a4f6c3377fa8c148dc7bea3457935d87c4b7457922" Oct 30 07:34:30.811185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.811157 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:30.927772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.927743 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\") pod \"8692621e-332b-47dc-9b56-ba297c3a7c95\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " Oct 30 07:34:30.927961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.927803 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\") pod \"8692621e-332b-47dc-9b56-ba297c3a7c95\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " Oct 30 07:34:30.928050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.927966 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\" (UniqueName: \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\") pod \"8692621e-332b-47dc-9b56-ba297c3a7c95\" (UID: \"8692621e-332b-47dc-9b56-ba297c3a7c95\") " Oct 30 07:34:30.928208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.928185 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca" pod "8692621e-332b-47dc-9b56-ba297c3a7c95" (UID: "8692621e-332b-47dc-9b56-ba297c3a7c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:30.928399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.928371 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:30.930062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.930039 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca" pod "8692621e-332b-47dc-9b56-ba297c3a7c95" (UID: "8692621e-332b-47dc-9b56-ba297c3a7c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:30.930159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:30.930135 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7" pod "8692621e-332b-47dc-9b56-ba297c3a7c95" (UID: "8692621e-332b-47dc-9b56-ba297c3a7c95"). InnerVolumeSpecName "kube-api-access-9gbp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:31.029341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.029315 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8692621e-332b-47dc-9b56-ba297c3a7c95-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:31.029483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.029344 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gbp7\" (UniqueName: \"kubernetes.io/projected/8692621e-332b-47dc-9b56-ba297c3a7c95-kube-api-access-9gbp7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:31.679799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.679766 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" Oct 30 07:34:31.679799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.679789 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc" event={"ID":"8692621e-332b-47dc-9b56-ba297c3a7c95","Type":"ContainerDied","Data":"af9dc55c63de9b8da9f60c3b7a329d27f4e245c71c5f9d6f44fcd90daccdcef7"} Oct 30 07:34:31.680243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.679837 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9dc55c63de9b8da9f60c3b7a329d27f4e245c71c5f9d6f44fcd90daccdcef7" Oct 30 07:34:31.832811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:31.832776 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:33.331735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:33.331696 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:33.442964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:33.442923 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:34.830432 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:34:34.830394 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:34:35.115904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:35.115823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:36.779938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:36.779903 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:37.331542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.331506 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5"] Oct 30 07:34:37.332300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332275 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8692621e-332b-47dc-9b56-ba297c3a7c95" containerName="revoke-keys" Oct 30 07:34:37.332300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332297 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8692621e-332b-47dc-9b56-ba297c3a7c95" containerName="revoke-keys" Oct 30 07:34:37.332481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332331 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ac5e2cc6-11c4-4960-ba19-90249a64d88e" containerName="revoke-keys" Oct 30 07:34:37.332481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332339 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5e2cc6-11c4-4960-ba19-90249a64d88e" containerName="revoke-keys" Oct 30 07:34:37.332481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332445 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5e2cc6-11c4-4960-ba19-90249a64d88e" containerName="revoke-keys" Oct 30 07:34:37.332481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.332463 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8692621e-332b-47dc-9b56-ba297c3a7c95" containerName="revoke-keys" Oct 30 07:34:37.336329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.336308 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.373697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.365068 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:34:37.373697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.365170 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:34:37.374696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.374656 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:34:37.374833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.374695 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:34:37.377105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.377056 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5"] Oct 30 07:34:37.415317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.415279 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.415531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.415484 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fkm\" (UniqueName: \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.415610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.415595 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.517030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.516986 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fkm\" (UniqueName: \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.517249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.517188 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.517410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.517345 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.518020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.518001 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.519582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.519563 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.540744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.540714 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fkm\" (UniqueName: \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\") pod \"maas-api-revoke-sub-b9e2d77f-lh4h5\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.652857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.652767 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:37.827107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.827068 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9"] Oct 30 07:34:37.832357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.832336 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:37.852426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.852393 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9"] Oct 30 07:34:37.930237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.930125 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:37.930411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.930358 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprng\" (UniqueName: \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:37.930518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.930495 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:37.961264 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:34:37.961232 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99718b6_8ba5_498a_8eca_2bd952168174.slice/crio-a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c WatchSource:0}: Error finding container a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c: Status 404 returned error can't find the container with id a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c Oct 30 07:34:37.961850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:37.961823 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5"] Oct 30 07:34:38.032315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.032279 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.032512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.032420 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprng\" (UniqueName: \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.032588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.032523 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.033197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.033173 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.034699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.034679 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.060644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.060611 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprng\" (UniqueName: \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\") pod \"maas-api-revoke-sub-8660fdf4-txvm9\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.146771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.146728 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:38.459141 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:34:38.459102 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660dc935_2e30_4bb6_8dd1_871f11baef20.slice/crio-e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb WatchSource:0}: Error finding container e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb: Status 404 returned error can't find the container with id e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb Oct 30 07:34:38.460025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.459876 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9"] Oct 30 07:34:38.529322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.529286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:38.742429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.742342 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" event={"ID":"b99718b6-8ba5-498a-8eca-2bd952168174","Type":"ContainerStarted","Data":"8903627e6b1da5f2893a860c9d5635508cd5f6e3805e6f8f4d4e364a0b25b026"} Oct 30 07:34:38.742429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.742391 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" event={"ID":"b99718b6-8ba5-498a-8eca-2bd952168174","Type":"ContainerStarted","Data":"a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c"} Oct 30 07:34:38.743780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.743753 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" event={"ID":"660dc935-2e30-4bb6-8dd1-871f11baef20","Type":"ContainerStarted","Data":"e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb"} Oct 30 07:34:38.786189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:38.786140 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" podStartSLOduration=1.786126228 podStartE2EDuration="1.786126228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:38.783551995 +0000 UTC m=+2242.474469431" watchObservedRunningTime="2026-10-30 07:34:38.786126228 +0000 UTC m=+2242.477043691" Oct 30 07:34:39.752073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:39.751989 5894 generic.go:356] "Generic (PLEG): container finished" podID="b99718b6-8ba5-498a-8eca-2bd952168174" containerID="8903627e6b1da5f2893a860c9d5635508cd5f6e3805e6f8f4d4e364a0b25b026" exitCode=0 Oct 30 07:34:39.752495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:39.752085 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" event={"ID":"b99718b6-8ba5-498a-8eca-2bd952168174","Type":"ContainerDied","Data":"8903627e6b1da5f2893a860c9d5635508cd5f6e3805e6f8f4d4e364a0b25b026"} Oct 30 07:34:39.753609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:39.753582 5894 generic.go:356] "Generic (PLEG): container finished" podID="660dc935-2e30-4bb6-8dd1-871f11baef20" containerID="5374d7b4b1f1ceb2090a6741ce08b80b2d0a2ba452930c4e2f75812129843d6d" exitCode=0 Oct 30 07:34:39.753770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:39.753672 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" event={"ID":"660dc935-2e30-4bb6-8dd1-871f11baef20","Type":"ContainerDied","Data":"5374d7b4b1f1ceb2090a6741ce08b80b2d0a2ba452930c4e2f75812129843d6d"} Oct 30 07:34:40.199505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:40.199470 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:40.971083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:40.971057 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:40.974679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:40.974623 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:41.078527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.078489 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\") pod \"660dc935-2e30-4bb6-8dd1-871f11baef20\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " Oct 30 07:34:41.078790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.078542 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\") pod \"b99718b6-8ba5-498a-8eca-2bd952168174\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " Oct 30 07:34:41.078790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.078714 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\" (UniqueName: \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\") pod \"b99718b6-8ba5-498a-8eca-2bd952168174\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " Oct 30 07:34:41.078790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.078758 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\" (UniqueName: \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\") pod \"660dc935-2e30-4bb6-8dd1-871f11baef20\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " Oct 30 07:34:41.079052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.079011 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca" pod "660dc935-2e30-4bb6-8dd1-871f11baef20" (UID: "660dc935-2e30-4bb6-8dd1-871f11baef20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:41.079052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.078816 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\") pod \"660dc935-2e30-4bb6-8dd1-871f11baef20\" (UID: \"660dc935-2e30-4bb6-8dd1-871f11baef20\") " Oct 30 07:34:41.079279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.079173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\") pod \"b99718b6-8ba5-498a-8eca-2bd952168174\" (UID: \"b99718b6-8ba5-498a-8eca-2bd952168174\") " Oct 30 07:34:41.079616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.079584 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca" pod "b99718b6-8ba5-498a-8eca-2bd952168174" (UID: "b99718b6-8ba5-498a-8eca-2bd952168174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:34:41.080049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.080025 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.080178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.080053 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.081418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.081381 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm" pod "b99718b6-8ba5-498a-8eca-2bd952168174" (UID: "b99718b6-8ba5-498a-8eca-2bd952168174"). InnerVolumeSpecName "kube-api-access-v5fkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:41.081747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.081720 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca" pod "660dc935-2e30-4bb6-8dd1-871f11baef20" (UID: "660dc935-2e30-4bb6-8dd1-871f11baef20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:41.081858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.081770 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng" pod "660dc935-2e30-4bb6-8dd1-871f11baef20" (UID: "660dc935-2e30-4bb6-8dd1-871f11baef20"). InnerVolumeSpecName "kube-api-access-jprng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:34:41.081858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.081775 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca" pod "b99718b6-8ba5-498a-8eca-2bd952168174" (UID: "b99718b6-8ba5-498a-8eca-2bd952168174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:34:41.181275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.181238 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b99718b6-8ba5-498a-8eca-2bd952168174-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.181275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.181270 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fkm\" (UniqueName: \"kubernetes.io/projected/b99718b6-8ba5-498a-8eca-2bd952168174-kube-api-access-v5fkm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.181275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.181279 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jprng\" (UniqueName: \"kubernetes.io/projected/660dc935-2e30-4bb6-8dd1-871f11baef20-kube-api-access-jprng\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.181610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.181288 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660dc935-2e30-4bb6-8dd1-871f11baef20-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:34:41.771348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.771317 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" Oct 30 07:34:41.771348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.771328 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5" event={"ID":"b99718b6-8ba5-498a-8eca-2bd952168174","Type":"ContainerDied","Data":"a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c"} Oct 30 07:34:41.771587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.771375 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c9af7c8195b26016871b3050e14ba6a5c86b9de3e755e5a16d9db535b24e6c" Oct 30 07:34:41.772922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.772884 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" Oct 30 07:34:41.773045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.772902 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9" event={"ID":"660dc935-2e30-4bb6-8dd1-871f11baef20","Type":"ContainerDied","Data":"e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb"} Oct 30 07:34:41.773045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.773012 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e80f318514161fe5dca8cad4a4c3e4e0d9258cd486908dcbf8830c339e7b63cb" Oct 30 07:34:41.903422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:41.903394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:43.511232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:43.511197 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:45.144344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:45.144306 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:46.834121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:46.834089 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:48.541104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:48.541064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:49.471565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:49.471517 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:34:49.530334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:49.530300 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:50.232864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:50.232827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:50.355285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:50.355243 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:50.570688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:50.570621 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:51.993222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:51.993188 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:53.054205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:53.054172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:53.600876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:53.600846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:54.244462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:54.244420 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:34:55.365702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:55.365655 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:55.519616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:55.519575 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:34:55.570011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:55.569975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:56.370427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.370394 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj"] Oct 30 07:34:56.371220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371199 5894 cpu_manager.go:414] "containerMap: removing container" podUID="660dc935-2e30-4bb6-8dd1-871f11baef20" containerName="revoke-keys" Oct 30 07:34:56.371374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371221 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660dc935-2e30-4bb6-8dd1-871f11baef20" containerName="revoke-keys" Oct 30 07:34:56.371374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371238 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b99718b6-8ba5-498a-8eca-2bd952168174" containerName="revoke-keys" Oct 30 07:34:56.371374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371246 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99718b6-8ba5-498a-8eca-2bd952168174" containerName="revoke-keys" Oct 30 07:34:56.371533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371480 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="660dc935-2e30-4bb6-8dd1-871f11baef20" containerName="revoke-keys" Oct 30 07:34:56.371533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.371496 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99718b6-8ba5-498a-8eca-2bd952168174" containerName="revoke-keys" Oct 30 07:34:56.376079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.376058 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.400333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.400295 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj"] Oct 30 07:34:56.400333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.400316 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:34:56.412144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.411974 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:34:56.412144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.411999 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:34:56.412144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.412109 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:34:56.462622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.462579 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtfq\" (UniqueName: \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.462828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.462662 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.462896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.462834 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.563867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.563831 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.564052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.564030 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtfq\" (UniqueName: \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.564097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.564058 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.564666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.564605 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.566711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.566688 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.587892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.587851 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtfq\" (UniqueName: \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\") pod \"maas-api-revoke-sub-a3ac01c3-hhswj\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.690999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.690903 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:34:56.998967 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:34:56.998933 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e28d84_f243_479f_9403_6fb9e0818912.slice/crio-f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484 WatchSource:0}: Error finding container f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484: Status 404 returned error can't find the container with id f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484 Oct 30 07:34:56.999957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:56.999926 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj"] Oct 30 07:34:57.248625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:57.248494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:34:57.899415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:57.899374 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" event={"ID":"d4e28d84-f243-479f-9403-6fb9e0818912","Type":"ContainerStarted","Data":"0d78f8db5fe82d12081913b07fbdc8218dec554a281a1f41b31c1afb6f427ea2"} Oct 30 07:34:57.899846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:57.899429 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" event={"ID":"d4e28d84-f243-479f-9403-6fb9e0818912","Type":"ContainerStarted","Data":"f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484"} Oct 30 07:34:57.946843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:57.946791 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" podStartSLOduration=1.946776383 podStartE2EDuration="1.946776383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:34:57.946141948 +0000 UTC m=+2261.637059386" watchObservedRunningTime="2026-10-30 07:34:57.946776383 +0000 UTC m=+2261.637693798" Oct 30 07:34:58.909285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:58.909248 5894 generic.go:356] "Generic (PLEG): container finished" podID="d4e28d84-f243-479f-9403-6fb9e0818912" containerID="0d78f8db5fe82d12081913b07fbdc8218dec554a281a1f41b31c1afb6f427ea2" exitCode=0 Oct 30 07:34:58.909756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:58.909296 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" event={"ID":"d4e28d84-f243-479f-9403-6fb9e0818912","Type":"ContainerDied","Data":"0d78f8db5fe82d12081913b07fbdc8218dec554a281a1f41b31c1afb6f427ea2"} Oct 30 07:34:59.080653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:34:59.080610 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:00.079765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.079742 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:35:00.134165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.134134 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\") pod \"d4e28d84-f243-479f-9403-6fb9e0818912\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " Oct 30 07:35:00.134365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.134196 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\" (UniqueName: \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\") pod \"d4e28d84-f243-479f-9403-6fb9e0818912\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " Oct 30 07:35:00.134365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.134310 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\") pod \"d4e28d84-f243-479f-9403-6fb9e0818912\" (UID: \"d4e28d84-f243-479f-9403-6fb9e0818912\") " Oct 30 07:35:00.134570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.134547 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca" pod "d4e28d84-f243-479f-9403-6fb9e0818912" (UID: "d4e28d84-f243-479f-9403-6fb9e0818912"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:00.134965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.134943 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:00.136344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.136317 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca" pod "d4e28d84-f243-479f-9403-6fb9e0818912" (UID: "d4e28d84-f243-479f-9403-6fb9e0818912"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:00.136442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.136411 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq" pod "d4e28d84-f243-479f-9403-6fb9e0818912" (UID: "d4e28d84-f243-479f-9403-6fb9e0818912"). InnerVolumeSpecName "kube-api-access-fvtfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:00.236436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.236359 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtfq\" (UniqueName: \"kubernetes.io/projected/d4e28d84-f243-479f-9403-6fb9e0818912-kube-api-access-fvtfq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:00.236436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.236389 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4e28d84-f243-479f-9403-6fb9e0818912-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:00.851144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.851112 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:00.926494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.926422 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" Oct 30 07:35:00.926494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.926464 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj" event={"ID":"d4e28d84-f243-479f-9403-6fb9e0818912","Type":"ContainerDied","Data":"f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484"} Oct 30 07:35:00.926810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:00.926534 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f772c8969738b2c2049941f88e39c074e3ac3444408601d68461eb3db3df3484" Oct 30 07:35:02.530753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:02.530723 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:03.215375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:03.215344 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:35:03.215584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:03.215530 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" containerID="cri-o://0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807" gracePeriod=30 Oct 30 07:35:04.240610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.240567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:04.507445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.507352 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" podUID="b8160f7c-ed30-4011-865d-d717c367c9fb" containerName="main" containerID="cri-o://8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767" gracePeriod=45 Oct 30 07:35:04.789370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.789337 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:35:04.846141 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:04.846096 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:35:04.899945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.899888 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\" (UniqueName: \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900061 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900096 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900153 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900260 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900314 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\") pod \"b8160f7c-ed30-4011-865d-d717c367c9fb\" (UID: \"b8160f7c-ed30-4011-865d-d717c367c9fb\") " Oct 30 07:35:04.900444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900377 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:04.900444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900377 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:04.900550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900496 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:04.900951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900919 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:04.900951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900951 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:04.901193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.900962 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:04.902482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.902458 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "kube-api-access-5gq79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:04.902668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.902613 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:04.902738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.902706 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm" pod "b8160f7c-ed30-4011-865d-d717c367c9fb" (UID: "b8160f7c-ed30-4011-865d-d717c367c9fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:04.958371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.958339 5894 generic.go:356] "Generic (PLEG): container finished" podID="b8160f7c-ed30-4011-865d-d717c367c9fb" containerID="8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767" exitCode=0 Oct 30 07:35:04.958524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.958401 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" Oct 30 07:35:04.958524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.958423 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" event={"ID":"b8160f7c-ed30-4011-865d-d717c367c9fb","Type":"ContainerDied","Data":"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767"} Oct 30 07:35:04.958524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.958467 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj" event={"ID":"b8160f7c-ed30-4011-865d-d717c367c9fb","Type":"ContainerDied","Data":"67e4edace722cc209a1a5c34b16b79f6f9995695add5a98caab0f56e4df38266"} Oct 30 07:35:04.958524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.958487 5894 scope.go:122] "RemoveContainer" containerID="8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767" Oct 30 07:35:04.969426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.969405 5894 scope.go:122] "RemoveContainer" containerID="8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767" Oct 30 07:35:04.969734 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:04.969712 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767\": container with ID starting with 8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767 not found: ID does not exist" containerID="8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767" Oct 30 07:35:04.969823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:04.969746 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767"} err="failed to get container status \"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767\": rpc error: code = NotFound desc = could not find container \"8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767\": container with ID starting with 8010a90cd9e56e6df693b9dc817b9f9fe00ff393fdb0b575aeeb3e5b526a3767 not found: ID does not exist" Oct 30 07:35:05.002096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.002066 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8160f7c-ed30-4011-865d-d717c367c9fb-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:05.002096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.002092 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8160f7c-ed30-4011-865d-d717c367c9fb-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:05.002096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.002101 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gq79\" (UniqueName: \"kubernetes.io/projected/b8160f7c-ed30-4011-865d-d717c367c9fb-kube-api-access-5gq79\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:05.011847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.011809 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:35:05.015574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.015508 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/auth-test-model-2e8b2a-kserve-7d849bdc6f-bqbmj"] Oct 30 07:35:05.159927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.156079 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:35:05.208284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.205670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:05.325676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.325606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:05.769815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.769113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:05.942664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:05.938997 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:06.190083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.190043 5894 patch_prober.go:29] interesting pod/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:35:06.190291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.190123 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:35:06.208118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.208084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:06.434415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.434380 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8160f7c-ed30-4011-865d-d717c367c9fb" path="/var/lib/kubelet/pods/b8160f7c-ed30-4011-865d-d717c367c9fb/volumes" Oct 30 07:35:06.704276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704230 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg"] Oct 30 07:35:06.704852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704836 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d4e28d84-f243-479f-9403-6fb9e0818912" containerName="revoke-keys" Oct 30 07:35:06.704852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704852 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e28d84-f243-479f-9403-6fb9e0818912" containerName="revoke-keys" Oct 30 07:35:06.705028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704866 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b8160f7c-ed30-4011-865d-d717c367c9fb" containerName="main" Oct 30 07:35:06.705028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704871 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8160f7c-ed30-4011-865d-d717c367c9fb" containerName="main" Oct 30 07:35:06.705028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704974 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8160f7c-ed30-4011-865d-d717c367c9fb" containerName="main" Oct 30 07:35:06.705028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.704993 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e28d84-f243-479f-9403-6fb9e0818912" containerName="revoke-keys" Oct 30 07:35:06.709971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.709947 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.729047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.729019 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg"] Oct 30 07:35:06.737467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.737441 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:35:06.737782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.737751 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:35:06.737900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.737811 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:35:06.738099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.738084 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:35:06.839942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.839908 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fg2r\" (UniqueName: \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.840146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.839973 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.840146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.840074 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.941360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.941320 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fg2r\" (UniqueName: \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.941581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.941406 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.941581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.941468 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.942376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.942353 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.944411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.944378 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:06.965145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:06.965059 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fg2r\" (UniqueName: \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\") pod \"maas-api-revoke-sub-02f5f5b5-f6bvg\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:07.025779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:07.025734 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:07.341176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:07.341146 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg"] Oct 30 07:35:07.341976 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:07.341945 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e41f78_c6f2_4e1b_bb3c_93712080dd92.slice/crio-fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e WatchSource:0}: Error finding container fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e: Status 404 returned error can't find the container with id fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e Oct 30 07:35:08.007130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.007027 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" event={"ID":"52e41f78-c6f2-4e1b-bb3c-93712080dd92","Type":"ContainerStarted","Data":"360812004f295844d9cd12dc4d039d1ceca1341ced4ea69443dfafcf8a0f98cc"} Oct 30 07:35:08.007130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.007093 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" event={"ID":"52e41f78-c6f2-4e1b-bb3c-93712080dd92","Type":"ContainerStarted","Data":"fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e"} Oct 30 07:35:08.051320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.051269 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" podStartSLOduration=2.051254914 podStartE2EDuration="2.051254914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:08.047905693 +0000 UTC m=+2271.738823130" watchObservedRunningTime="2026-10-30 07:35:08.051254914 +0000 UTC m=+2271.742172364" Oct 30 07:35:08.056197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.056171 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:08.498891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.498862 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:35:08.678904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.678819 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.678924 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.678987 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679059 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679115 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679207 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:08.679253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679215 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679301 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679355 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679412 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\") pod \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\" (UID: \"e7668dbc-9a31-4f5d-9c2b-317a5c024375\") " Oct 30 07:35:08.679471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679401 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:08.679685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679656 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:08.679791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.679741 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:08.680079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.680059 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.680174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.680085 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.680174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.680097 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.680174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.680105 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.680174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.680056 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:08.681179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.681158 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:08.681534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.681518 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:08.681690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.681663 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:08.681782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.681744 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz" pod "e7668dbc-9a31-4f5d-9c2b-317a5c024375" (UID: "e7668dbc-9a31-4f5d-9c2b-317a5c024375"). InnerVolumeSpecName "kube-api-access-254dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:08.781408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.781374 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.781408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.781403 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.781408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.781413 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.781666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.781423 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254dz\" (UniqueName: \"kubernetes.io/projected/e7668dbc-9a31-4f5d-9c2b-317a5c024375-kube-api-access-254dz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:08.781666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:08.781433 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7668dbc-9a31-4f5d-9c2b-317a5c024375-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:09.017941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.017901 5894 generic.go:356] "Generic (PLEG): container finished" podID="52e41f78-c6f2-4e1b-bb3c-93712080dd92" containerID="360812004f295844d9cd12dc4d039d1ceca1341ced4ea69443dfafcf8a0f98cc" exitCode=0 Oct 30 07:35:09.018426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.017985 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" event={"ID":"52e41f78-c6f2-4e1b-bb3c-93712080dd92","Type":"ContainerDied","Data":"360812004f295844d9cd12dc4d039d1ceca1341ced4ea69443dfafcf8a0f98cc"} Oct 30 07:35:09.019558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.019530 5894 generic.go:356] "Generic (PLEG): container finished" podID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerID="0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807" exitCode=0 Oct 30 07:35:09.019654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.019574 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" Oct 30 07:35:09.019654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.019593 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" event={"ID":"e7668dbc-9a31-4f5d-9c2b-317a5c024375","Type":"ContainerDied","Data":"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807"} Oct 30 07:35:09.019736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.019662 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq" event={"ID":"e7668dbc-9a31-4f5d-9c2b-317a5c024375","Type":"ContainerDied","Data":"7de10756277e4e43ebd9fed4196283fbe0f3d0d961d767db64aa683a17166640"} Oct 30 07:35:09.019736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.019686 5894 scope.go:122] "RemoveContainer" containerID="0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807" Oct 30 07:35:09.030703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.030681 5894 scope.go:122] "RemoveContainer" containerID="0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807" Oct 30 07:35:09.030982 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:09.030963 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807\": container with ID starting with 0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807 not found: ID does not exist" containerID="0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807" Oct 30 07:35:09.031048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.030991 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807"} err="failed to get container status \"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807\": rpc error: code = NotFound desc = could not find container \"0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807\": container with ID starting with 0826b157a61191c3c5e644010c4455a6edfd97f5892966fe0d5dbf01167a2807 not found: ID does not exist" Oct 30 07:35:09.109896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.109862 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:35:09.113357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.113332 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b72d79-openshift-default-c747b9bc9-9pzwq"] Oct 30 07:35:09.359442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.359407 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:09.479691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.479658 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:09.940601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:09.940560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:10.196483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.196415 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:10.303992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.303953 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\" (UniqueName: \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\") pod \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " Oct 30 07:35:10.304200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.303999 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\") pod \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " Oct 30 07:35:10.304200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.304176 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\") pod \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\" (UID: \"52e41f78-c6f2-4e1b-bb3c-93712080dd92\") " Oct 30 07:35:10.304561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.304535 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca" pod "52e41f78-c6f2-4e1b-bb3c-93712080dd92" (UID: "52e41f78-c6f2-4e1b-bb3c-93712080dd92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:10.304718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.304676 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.306143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.306117 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r" pod "52e41f78-c6f2-4e1b-bb3c-93712080dd92" (UID: "52e41f78-c6f2-4e1b-bb3c-93712080dd92"). InnerVolumeSpecName "kube-api-access-8fg2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:10.306237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.306203 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca" pod "52e41f78-c6f2-4e1b-bb3c-93712080dd92" (UID: "52e41f78-c6f2-4e1b-bb3c-93712080dd92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:10.406408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.406372 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fg2r\" (UniqueName: \"kubernetes.io/projected/52e41f78-c6f2-4e1b-bb3c-93712080dd92-kube-api-access-8fg2r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.406408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.406405 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52e41f78-c6f2-4e1b-bb3c-93712080dd92-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.432822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.432788 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" path="/var/lib/kubelet/pods/e7668dbc-9a31-4f5d-9c2b-317a5c024375/volumes" Oct 30 07:35:10.551004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.550963 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" podUID="8e2fe803-a207-49f5-90be-67c85d293167" containerName="main" containerID="cri-o://427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa" gracePeriod=45 Oct 30 07:35:10.712551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.712513 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" podUID="8e2fe803-a207-49f5-90be-67c85d293167" containerName="main" probeResult="failure" output="Get \"https://10.134.0.193:8000/ready\": dial tcp 10.134.0.193:8000: connect: connection refused" Oct 30 07:35:10.811382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.811358 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:35:10.911492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911453 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\" (UniqueName: \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911506 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911530 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911651 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911686 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911720 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\") pod \"8e2fe803-a207-49f5-90be-67c85d293167\" (UID: \"8e2fe803-a207-49f5-90be-67c85d293167\") " Oct 30 07:35:10.911878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911851 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:10.911878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911868 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:10.912068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.911994 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:10.912294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.912270 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.912294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.912296 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.912477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.912306 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:10.913953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.913933 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:10.914035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.913976 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:10.914035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:10.914026 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt" pod "8e2fe803-a207-49f5-90be-67c85d293167" (UID: "8e2fe803-a207-49f5-90be-67c85d293167"). InnerVolumeSpecName "kube-api-access-mftqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:11.013152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.013117 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e2fe803-a207-49f5-90be-67c85d293167-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:11.013152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.013149 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e2fe803-a207-49f5-90be-67c85d293167-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:11.013152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.013159 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftqt\" (UniqueName: \"kubernetes.io/projected/8e2fe803-a207-49f5-90be-67c85d293167-kube-api-access-mftqt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:11.036350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.036318 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" Oct 30 07:35:11.036560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.036351 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg" event={"ID":"52e41f78-c6f2-4e1b-bb3c-93712080dd92","Type":"ContainerDied","Data":"fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e"} Oct 30 07:35:11.036560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.036416 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb63a944ff6b156b7ade33b8cf209dd36fc6e7b279b1a80832afa67cb543dd1e" Oct 30 07:35:11.037932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.037904 5894 generic.go:356] "Generic (PLEG): container finished" podID="8e2fe803-a207-49f5-90be-67c85d293167" containerID="427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa" exitCode=0 Oct 30 07:35:11.038052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.037947 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" Oct 30 07:35:11.038052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.037989 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" event={"ID":"8e2fe803-a207-49f5-90be-67c85d293167","Type":"ContainerDied","Data":"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa"} Oct 30 07:35:11.038052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.038033 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq" event={"ID":"8e2fe803-a207-49f5-90be-67c85d293167","Type":"ContainerDied","Data":"dc7d628c645ee53c5c0395e8cea0b1c8c8dd88b500ddb1307348b75e1888c4d6"} Oct 30 07:35:11.038188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.038054 5894 scope.go:122] "RemoveContainer" containerID="427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa" Oct 30 07:35:11.048932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.048845 5894 scope.go:122] "RemoveContainer" containerID="427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa" Oct 30 07:35:11.049493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.049472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:11.052816 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:11.052758 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa\": container with ID starting with 427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa not found: ID does not exist" containerID="427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa" Oct 30 07:35:11.052816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.052792 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa"} err="failed to get container status \"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa\": rpc error: code = NotFound desc = could not find container \"427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa\": container with ID starting with 427cb0d13707fd16ad42eb38c18c8b47666aa0bc73d207d5400e502177dffffa not found: ID does not exist" Oct 30 07:35:11.095459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.095413 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:35:11.105564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.105533 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-4eff1f/auth-test-model-4eff1f-kserve-87b9bddf7-fxhlq"] Oct 30 07:35:11.174160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.174121 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:11.724321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:11.724283 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:12.432491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:12.432456 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2fe803-a207-49f5-90be-67c85d293167" path="/var/lib/kubelet/pods/8e2fe803-a207-49f5-90be-67c85d293167/volumes" Oct 30 07:35:13.505828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:13.505789 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:15.258869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:15.258836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:16.931835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:16.931800 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:18.032327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.032295 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn"] Oct 30 07:35:18.044820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.044793 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88885886-bd9gn"] Oct 30 07:35:18.467021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.466530 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df455f56-b4ac-451a-a3e7-7ff020c6f3a8" path="/var/lib/kubelet/pods/df455f56-b4ac-451a-a3e7-7ff020c6f3a8/volumes" Oct 30 07:35:18.736608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.736528 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:18.741295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.741267 5894 scope.go:122] "RemoveContainer" containerID="9d4c65d9e0aedca2c98f2902ed9bb9293504dc0022aae8fbf6408cb04e8af93e" Oct 30 07:35:18.751836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.751812 5894 scope.go:122] "RemoveContainer" containerID="b0721b496d379806fbe77013a2985c42b4873e11296528c047c04e59b75e9bfe" Oct 30 07:35:18.764461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.764435 5894 scope.go:122] "RemoveContainer" containerID="87221f6cfadd74d0d4ae9ebcef7d57761bb4148c1dbf9b46adf9515e628b8ae7" Oct 30 07:35:18.800782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.800646 5894 scope.go:122] "RemoveContainer" containerID="8cdf61859eba48c1b06e2d8645d63e5679574170226edf5a0c8621be5bdb34ef" Oct 30 07:35:18.815811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.815626 5894 scope.go:122] "RemoveContainer" containerID="ed10126e6309d370aa6750db67a926608a0b6ccc0a706fefda71778dc0f09649" Oct 30 07:35:18.826858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:18.826802 5894 scope.go:122] "RemoveContainer" containerID="d79b4798091f154d5512844f87afdc743c639adb7c00a41661e1bebfdcbbf7c3" Oct 30 07:35:20.182874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.182813 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerName="main" containerID="cri-o://bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c" gracePeriod=45 Oct 30 07:35:20.441902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.441871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:20.458156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.458124 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:35:20.558792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.558759 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.558826 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.558871 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.558969 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\" (UniqueName: \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559017 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559016 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:20.559191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559074 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\") pod \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\" (UID: \"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0\") " Oct 30 07:35:20.559358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559261 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:20.559358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559328 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:20.559769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559748 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:20.559837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559775 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:20.559837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.559790 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:20.561171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.561147 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:20.561285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.561153 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:20.561285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.561234 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp" pod "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" (UID: "8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0"). InnerVolumeSpecName "kube-api-access-tjpwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:20.660943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.660894 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:20.660943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.660928 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpwp\" (UniqueName: \"kubernetes.io/projected/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-kube-api-access-tjpwp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:20.660943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:20.660947 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:21.114816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.114783 5894 generic.go:356] "Generic (PLEG): container finished" podID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerID="bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c" exitCode=0 Oct 30 07:35:21.115037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.114843 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" Oct 30 07:35:21.115037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.114869 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" event={"ID":"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0","Type":"ContainerDied","Data":"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c"} Oct 30 07:35:21.115037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.114919 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" event={"ID":"8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0","Type":"ContainerDied","Data":"b5639220381d7ef340cc57c07a1de5f19ef930458e00412f45a510b198183782"} Oct 30 07:35:21.115037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.114945 5894 scope.go:122] "RemoveContainer" containerID="bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c" Oct 30 07:35:21.126199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.126182 5894 scope.go:122] "RemoveContainer" containerID="bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c" Oct 30 07:35:21.126481 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:21.126465 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c\": container with ID starting with bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c not found: ID does not exist" containerID="bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c" Oct 30 07:35:21.126540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.126489 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c"} err="failed to get container status \"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c\": rpc error: code = NotFound desc = could not find container \"bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c\": container with ID starting with bd6c062e0e35fd2c6180f56e8265e538bd4ebc8d6fe8fd9371a0a5ecc5f7fb0c not found: ID does not exist" Oct 30 07:35:21.167712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.167675 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:35:21.180010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.179974 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s"] Oct 30 07:35:21.448874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:21.448775 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-2761aa/auth-test-model-2761aa-kserve-59b4b76897-bdt5s" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.191:8000/ready\": context deadline exceeded" Oct 30 07:35:22.096490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:22.096447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:22.224114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:22.224078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:22.436333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:22.436245 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" path="/var/lib/kubelet/pods/8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0/volumes" Oct 30 07:35:22.555339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:22.555298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:23.027031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:23.026986 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:23.144515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:23.144469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:23.882490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:23.882438 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:24.709751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:24.709713 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:35:24.740285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:24.740250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:25.911989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:25.911952 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:26.327019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:26.326976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:26.782146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:26.782106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:26.937956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:26.937922 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:27.139400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:27.139315 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:27.218849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:27.218817 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:35:27.277856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:27.277821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:27.978998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:27.978957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:29.312152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:29.312115 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:29.460677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:29.460584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:29.782656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:29.781681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:29.911752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:29.911722 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:30.624392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:30.624360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:30.773280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:30.773246 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:31.757516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:31.757478 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:32.318606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.318577 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:32.436956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.436920 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:32.520841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.520804 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 07:35:32.521386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521371 5894 cpu_manager.go:414] "containerMap: removing container" podUID="52e41f78-c6f2-4e1b-bb3c-93712080dd92" containerName="revoke-keys" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521388 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e41f78-c6f2-4e1b-bb3c-93712080dd92" containerName="revoke-keys" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521404 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521410 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521442 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerName="main" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521446 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerName="main" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521460 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8e2fe803-a207-49f5-90be-67c85d293167" containerName="main" Oct 30 07:35:32.521469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521465 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2fe803-a207-49f5-90be-67c85d293167" containerName="main" Oct 30 07:35:32.521751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521577 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2fe803-a207-49f5-90be-67c85d293167" containerName="main" Oct 30 07:35:32.521751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521588 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e41f78-c6f2-4e1b-bb3c-93712080dd92" containerName="revoke-keys" Oct 30 07:35:32.521751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521594 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7668dbc-9a31-4f5d-9c2b-317a5c024375" containerName="istio-proxy" Oct 30 07:35:32.521751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.521604 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdaf9f9-53f9-4cc7-9b1a-de4f5eee35e0" containerName="main" Oct 30 07:35:32.526763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.526738 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.554769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.554738 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-705337-dockercfg-fx6lk\"" Oct 30 07:35:32.554769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.554738 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-705337\"" Oct 30 07:35:32.557610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.557583 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 07:35:32.648376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.648285 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x8hg\" (UniqueName: \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.648784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.648759 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.658665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.658339 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 07:35:32.667088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.667058 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.715250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.715196 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 07:35:32.750022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.749987 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfg9\" (UniqueName: \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.750222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.750188 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.750271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.750238 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x8hg\" (UniqueName: \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.750428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.750408 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.751254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.751234 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.774087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.774052 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x8hg\" (UniqueName: \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\") pod \"payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.841377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.841340 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:32.852139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.852107 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfg9\" (UniqueName: \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.852306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.852254 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.853126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.853100 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:32.876093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:32.876059 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfg9\" (UniqueName: \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\") pod \"payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:33.003471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.003429 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:33.149759 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:33.149613 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89a1b98c_1783_423c_9ed0_16b7ae07d9f5.slice/crio-dc25ca64def56d53111cdaa097fe6f700b514d8b8eaf745cefb175b2cbf69cb3 WatchSource:0}: Error finding container dc25ca64def56d53111cdaa097fe6f700b514d8b8eaf745cefb175b2cbf69cb3: Status 404 returned error can't find the container with id dc25ca64def56d53111cdaa097fe6f700b514d8b8eaf745cefb175b2cbf69cb3 Oct 30 07:35:33.151416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.151390 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 07:35:33.205987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.205956 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" event={"ID":"89a1b98c-1783-423c-9ed0-16b7ae07d9f5","Type":"ContainerStarted","Data":"dc25ca64def56d53111cdaa097fe6f700b514d8b8eaf745cefb175b2cbf69cb3"} Oct 30 07:35:33.312414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.312331 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 07:35:33.312689 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:33.312617 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d4057f_3238_4f1a_96fb_b28345420717.slice/crio-8ec1c64457ca0ec2f9b40f73a56a153cf7d7530f08b04732738159dff524ee83 WatchSource:0}: Error finding container 8ec1c64457ca0ec2f9b40f73a56a153cf7d7530f08b04732738159dff524ee83: Status 404 returned error can't find the container with id 8ec1c64457ca0ec2f9b40f73a56a153cf7d7530f08b04732738159dff524ee83 Oct 30 07:35:33.549558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.549525 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:33.592127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.592086 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:35:33.592373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:33.592347 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" containerID="cri-o://6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a" gracePeriod=30 Oct 30 07:35:34.213595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.213554 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" event={"ID":"89a1b98c-1783-423c-9ed0-16b7ae07d9f5","Type":"ContainerStarted","Data":"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435"} Oct 30 07:35:34.214120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.213749 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:34.215329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.215293 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" event={"ID":"83d4057f-3238-4f1a-96fb-b28345420717","Type":"ContainerStarted","Data":"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba"} Oct 30 07:35:34.215453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.215357 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" event={"ID":"83d4057f-3238-4f1a-96fb-b28345420717","Type":"ContainerStarted","Data":"8ec1c64457ca0ec2f9b40f73a56a153cf7d7530f08b04732738159dff524ee83"} Oct 30 07:35:34.215453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.215384 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:34.256358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.256304 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" podStartSLOduration=2.256286013 podStartE2EDuration="2.256286013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:34.254831627 +0000 UTC m=+2297.945749064" watchObservedRunningTime="2026-10-30 07:35:34.256286013 +0000 UTC m=+2297.947203451" Oct 30 07:35:34.298672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:34.298579 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" podStartSLOduration=2.298558759 podStartE2EDuration="2.298558759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:34.296518558 +0000 UTC m=+2297.987435995" watchObservedRunningTime="2026-10-30 07:35:34.298558759 +0000 UTC m=+2297.989476196" Oct 30 07:35:34.830221 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:34.830186 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:35:35.240085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:35.239977 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:36.782980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:36.782931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:36.921904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:36.921866 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:37.033283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:37.033198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:37.354684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:37.354581 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:35:38.745653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:38.745607 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:38.890488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:38.890414 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:35:39.052968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.052925 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.052977 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053051 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053111 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053134 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053267 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053384 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053444 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:39.053595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053538 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:39.053595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053545 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:35:39.053595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053542 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053628 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\") pod \"631af43a-06eb-4902-ab4f-18a0134d2dae\" (UID: \"631af43a-06eb-4902-ab4f-18a0134d2dae\") " Oct 30 07:35:39.053845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053788 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:39.053933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.053904 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:39.054349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.054326 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/631af43a-06eb-4902-ab4f-18a0134d2dae-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.054423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.054356 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.054423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.054371 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.054423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.054385 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.054423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.054399 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.055531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.055508 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "kube-api-access-cm7zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:39.055697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.055667 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:39.055866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.055837 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:35:39.055953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.055855 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token" pod "631af43a-06eb-4902-ab4f-18a0134d2dae" (UID: "631af43a-06eb-4902-ab4f-18a0134d2dae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:39.156107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.156025 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm7zd\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-kube-api-access-cm7zd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.156107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.156052 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.156107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.156062 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.156107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.156070 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/631af43a-06eb-4902-ab4f-18a0134d2dae-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:39.261599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.261561 5894 generic.go:356] "Generic (PLEG): container finished" podID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerID="6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a" exitCode=0 Oct 30 07:35:39.261834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.261615 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" Oct 30 07:35:39.261834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.261678 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" event={"ID":"631af43a-06eb-4902-ab4f-18a0134d2dae","Type":"ContainerDied","Data":"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a"} Oct 30 07:35:39.261834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.261735 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9" event={"ID":"631af43a-06eb-4902-ab4f-18a0134d2dae","Type":"ContainerDied","Data":"f96b185d1342247d52b90cf30b1b5fc45d96ebc7bad18380b791b458f72547e7"} Oct 30 07:35:39.261834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.261758 5894 scope.go:122] "RemoveContainer" containerID="6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a" Oct 30 07:35:39.272858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.272835 5894 scope.go:122] "RemoveContainer" containerID="6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a" Oct 30 07:35:39.273162 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:39.273144 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a\": container with ID starting with 6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a not found: ID does not exist" containerID="6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a" Oct 30 07:35:39.273216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.273171 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a"} err="failed to get container status \"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a\": rpc error: code = NotFound desc = could not find container \"6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a\": container with ID starting with 6c2c8244d0c6b1a2a31571cd96d0ac05c9eff1d0734084daf7e4b19c1f440e9a not found: ID does not exist" Oct 30 07:35:39.315905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.315870 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:35:39.320281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.320255 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-g7kh9"] Oct 30 07:35:39.739452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.739406 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" containerName="main" containerID="cri-o://724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9" gracePeriod=45 Oct 30 07:35:39.844279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:39.844229 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" containerName="main" probeResult="failure" output="Get \"https://10.134.0.195:8000/ready\": dial tcp 10.134.0.195:8000: connect: connection refused" Oct 30 07:35:40.026411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.026375 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:35:40.168977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.168942 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169009 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\" (UniqueName: \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169111 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\" (UniqueName: \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169159 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169296 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169342 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\") pod \"74dbc724-84a7-4263-a1f2-8224143fe224\" (UID: \"74dbc724-84a7-4263-a1f2-8224143fe224\") " Oct 30 07:35:40.169475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169455 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:40.169593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169564 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:40.169811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.169785 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:40.170031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.170008 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.170031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.170032 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.170223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.170043 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.171419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.171393 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:40.171514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.171488 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:40.174056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.174034 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb" pod "74dbc724-84a7-4263-a1f2-8224143fe224" (UID: "74dbc724-84a7-4263-a1f2-8224143fe224"). InnerVolumeSpecName "kube-api-access-5xxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:40.269519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.269431 5894 generic.go:356] "Generic (PLEG): container finished" podID="74dbc724-84a7-4263-a1f2-8224143fe224" containerID="724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9" exitCode=0 Oct 30 07:35:40.269519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.269477 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" Oct 30 07:35:40.269768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.269514 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" event={"ID":"74dbc724-84a7-4263-a1f2-8224143fe224","Type":"ContainerDied","Data":"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9"} Oct 30 07:35:40.269768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.269555 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh" event={"ID":"74dbc724-84a7-4263-a1f2-8224143fe224","Type":"ContainerDied","Data":"d68499ceab564b9dea62ccaf9d2c63007bb05155dab4bfdd63c4a73653ab38a8"} Oct 30 07:35:40.269768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.269573 5894 scope.go:122] "RemoveContainer" containerID="724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9" Oct 30 07:35:40.270953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.270928 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74dbc724-84a7-4263-a1f2-8224143fe224-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.271090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.270956 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74dbc724-84a7-4263-a1f2-8224143fe224-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.271090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.270970 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xxqb\" (UniqueName: \"kubernetes.io/projected/74dbc724-84a7-4263-a1f2-8224143fe224-kube-api-access-5xxqb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:40.284928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.284905 5894 scope.go:122] "RemoveContainer" containerID="724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9" Oct 30 07:35:40.285274 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:40.285251 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9\": container with ID starting with 724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9 not found: ID does not exist" containerID="724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9" Oct 30 07:35:40.285353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.285283 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9"} err="failed to get container status \"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9\": rpc error: code = NotFound desc = could not find container \"724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9\": container with ID starting with 724444b588536191f236d281b047af78073aaa7e0a1a4781e063133bd45784a9 not found: ID does not exist" Oct 30 07:35:40.322035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.322003 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:35:40.328288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.328259 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-4b0e81/auth-test-model-4b0e81-kserve-848b94fb45-m2jjh"] Oct 30 07:35:40.431778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.431743 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" path="/var/lib/kubelet/pods/631af43a-06eb-4902-ab4f-18a0134d2dae/volumes" Oct 30 07:35:40.432242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.432228 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" path="/var/lib/kubelet/pods/74dbc724-84a7-4263-a1f2-8224143fe224/volumes" Oct 30 07:35:40.508568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:40.508532 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:42.113695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.113662 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:42.246200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.246157 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" podUID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" containerName="main" containerID="cri-o://3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718" gracePeriod=45 Oct 30 07:35:42.417246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.417212 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:35:42.465861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.464569 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:42.689164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.689126 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:35:42.700802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.700770 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7602-9ca2bd"] Oct 30 07:35:42.701341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701328 5894 cpu_manager.go:414] "containerMap: removing container" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" containerName="main" Oct 30 07:35:42.701399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701342 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" containerName="main" Oct 30 07:35:42.701399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701357 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" containerName="main" Oct 30 07:35:42.701399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701363 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" containerName="main" Oct 30 07:35:42.701399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701398 5894 cpu_manager.go:414] "containerMap: removing container" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" Oct 30 07:35:42.701543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701404 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" Oct 30 07:35:42.701543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701495 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="631af43a-06eb-4902-ab4f-18a0134d2dae" containerName="istio-proxy" Oct 30 07:35:42.701543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701503 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="74dbc724-84a7-4263-a1f2-8224143fe224" containerName="main" Oct 30 07:35:42.701543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.701511 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" containerName="main" Oct 30 07:35:42.704963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.704940 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:42.707775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.707751 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\" (UniqueName: \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.707934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.707819 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.707934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.707870 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.707934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.707901 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.708123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.707947 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.708123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\") pod \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\" (UID: \"1d48065e-fd95-4a9e-a2b7-2f300bca75bd\") " Oct 30 07:35:42.708123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708109 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:42.708342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708316 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:42.708501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708472 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7s4x\" (UniqueName: \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\") pod \"test-curl-7602-9ca2bd\" (UID: \"99c48f78-bdd8-4c63-8285-55c8145448ac\") " pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:42.708629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708605 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.708629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.708621 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.710539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.710515 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:42.710693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.710675 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:42.710769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.710689 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:42.710939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.710915 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959" pod "1d48065e-fd95-4a9e-a2b7-2f300bca75bd" (UID: "1d48065e-fd95-4a9e-a2b7-2f300bca75bd"). InnerVolumeSpecName "kube-api-access-k6959". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:42.718946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.718914 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-9ca2bd"] Oct 30 07:35:42.720006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.719981 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:35:42.810315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.810269 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7s4x\" (UniqueName: \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\") pod \"test-curl-7602-9ca2bd\" (UID: \"99c48f78-bdd8-4c63-8285-55c8145448ac\") " pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:42.810497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.810387 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.810497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.810408 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6959\" (UniqueName: \"kubernetes.io/projected/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-kube-api-access-k6959\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.810497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.810422 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.810497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.810434 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d48065e-fd95-4a9e-a2b7-2f300bca75bd-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:42.833675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:42.833619 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7s4x\" (UniqueName: \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\") pod \"test-curl-7602-9ca2bd\" (UID: \"99c48f78-bdd8-4c63-8285-55c8145448ac\") " pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:43.062818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.062780 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:43.304828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.304793 5894 generic.go:356] "Generic (PLEG): container finished" podID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" containerID="3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718" exitCode=0 Oct 30 07:35:43.305275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.304857 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" Oct 30 07:35:43.305275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.304865 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" event={"ID":"1d48065e-fd95-4a9e-a2b7-2f300bca75bd","Type":"ContainerDied","Data":"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718"} Oct 30 07:35:43.305275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.304904 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w" event={"ID":"1d48065e-fd95-4a9e-a2b7-2f300bca75bd","Type":"ContainerDied","Data":"f0dd645baef28fe9f2fd52dfd59b80d1d90fd419b5e083911cb1623dce574762"} Oct 30 07:35:43.305275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.304927 5894 scope.go:122] "RemoveContainer" containerID="3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718" Oct 30 07:35:43.315204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.315180 5894 scope.go:122] "RemoveContainer" containerID="3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718" Oct 30 07:35:43.315534 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:43.315511 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718\": container with ID starting with 3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718 not found: ID does not exist" containerID="3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718" Oct 30 07:35:43.315630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.315541 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718"} err="failed to get container status \"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718\": rpc error: code = NotFound desc = could not find container \"3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718\": container with ID starting with 3afdd093e179d2af457505fae9d06edf24ee9514614d6d755c560b1b2b8fc718 not found: ID does not exist" Oct 30 07:35:43.362516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.362486 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:35:43.375695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.375668 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/auth-test-model-bf563a-kserve-6c6cd5bf6f-2f84w"] Oct 30 07:35:43.377185 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:43.377158 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99c48f78_bdd8_4c63_8285_55c8145448ac.slice/crio-759b8d46b1116f9e7fd3154949e37868f109e1c32a704622ec42b0fa6e11ed23 WatchSource:0}: Error finding container 759b8d46b1116f9e7fd3154949e37868f109e1c32a704622ec42b0fa6e11ed23: Status 404 returned error can't find the container with id 759b8d46b1116f9e7fd3154949e37868f109e1c32a704622ec42b0fa6e11ed23 Oct 30 07:35:43.379136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.379086 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7602-9ca2bd"] Oct 30 07:35:43.813814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:43.813781 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:44.312884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:44.312849 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-9ca2bd" event={"ID":"99c48f78-bdd8-4c63-8285-55c8145448ac","Type":"ContainerStarted","Data":"ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89"} Oct 30 07:35:44.313380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:44.312899 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7602-9ca2bd" event={"ID":"99c48f78-bdd8-4c63-8285-55c8145448ac","Type":"ContainerStarted","Data":"759b8d46b1116f9e7fd3154949e37868f109e1c32a704622ec42b0fa6e11ed23"} Oct 30 07:35:44.353484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:44.353430 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7602-9ca2bd" podStartSLOduration=2.353414126 podStartE2EDuration="2.353414126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:44.351279463 +0000 UTC m=+2308.042196899" watchObservedRunningTime="2026-10-30 07:35:44.353414126 +0000 UTC m=+2308.044331563" Oct 30 07:35:44.432968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:44.432936 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d48065e-fd95-4a9e-a2b7-2f300bca75bd" path="/var/lib/kubelet/pods/1d48065e-fd95-4a9e-a2b7-2f300bca75bd/volumes" Oct 30 07:35:45.214065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:45.214012 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7602-9ca2bd"] Oct 30 07:35:45.220320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:45.220287 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7602-9ca2bd"] Oct 30 07:35:45.231574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:45.231545 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 07:35:45.231769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:45.231621 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 07:35:45.528054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:45.528020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:46.327334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:46.327297 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7602-9ca2bd" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" containerName="test-curl-7602-9ca2bd" containerID="cri-o://ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89" gracePeriod=2 Oct 30 07:35:46.341220 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:46.341185 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-9ca2bd\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:46.441623 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:46.441581 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-9ca2bd\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:46.870067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:46.870037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:47.160300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:47.145985 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:47.494715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:47.493843 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:47.671031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:47.670857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:48.557576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:48.557546 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:48.586969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:48.586881 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\" (UniqueName: \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\") pod \"99c48f78-bdd8-4c63-8285-55c8145448ac\" (UID: \"99c48f78-bdd8-4c63-8285-55c8145448ac\") " Oct 30 07:35:48.589196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:48.589165 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x" pod "99c48f78-bdd8-4c63-8285-55c8145448ac" (UID: "99c48f78-bdd8-4c63-8285-55c8145448ac"). InnerVolumeSpecName "kube-api-access-x7s4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:48.689371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:48.689323 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7s4x\" (UniqueName: \"kubernetes.io/projected/99c48f78-bdd8-4c63-8285-55c8145448ac-kube-api-access-x7s4x\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:48.951988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:48.951906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:49.349711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:49.349676 5894 generic.go:356] "Generic (PLEG): container finished" podID="99c48f78-bdd8-4c63-8285-55c8145448ac" containerID="ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89" exitCode=137 Oct 30 07:35:49.349711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:49.349714 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:49.349967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:49.349767 5894 scope.go:122] "RemoveContainer" containerID="ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89" Oct 30 07:35:49.360167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:49.360150 5894 scope.go:122] "RemoveContainer" containerID="ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89" Oct 30 07:35:49.360509 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:49.360481 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89\": container with ID starting with ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89 not found: ID does not exist" containerID="ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89" Oct 30 07:35:49.360568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:49.360519 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89"} err="failed to get container status \"ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89\": rpc error: code = NotFound desc = could not find container \"ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89\": container with ID starting with ad394038d929e591acaf2db5e385a3967cdfdac5347acc223e1f0099a21cae89 not found: ID does not exist" Oct 30 07:35:49.372772 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:49.372729 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7602-9ca2bd\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" pod="opendatahub/test-curl-7602-9ca2bd" Oct 30 07:35:50.431347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:50.431312 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" path="/var/lib/kubelet/pods/99c48f78-bdd8-4c63-8285-55c8145448ac/volumes" Oct 30 07:35:50.851897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:50.847806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:50.996345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:50.996309 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:52.667195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.667156 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:52.783170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.783133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:52.831867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.831831 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 07:35:52.832459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.832443 5894 cpu_manager.go:414] "containerMap: removing container" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" containerName="test-curl-7602-9ca2bd" Oct 30 07:35:52.832509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.832462 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" containerName="test-curl-7602-9ca2bd" Oct 30 07:35:52.832662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.832650 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c48f78-bdd8-4c63-8285-55c8145448ac" containerName="test-curl-7602-9ca2bd" Oct 30 07:35:52.837904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.837882 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:52.853887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.853848 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-b096ec\"" Oct 30 07:35:52.866421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.866379 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-b096ec-dockercfg-p744q\"" Oct 30 07:35:52.869946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.869918 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 07:35:52.925231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.922279 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:52.945659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.944705 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 07:35:52.974655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.974058 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:52.974655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:52.974184 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lmc\" (UniqueName: \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.014212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.006103 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 07:35:53.014212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.006234 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.075930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.075891 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.076156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.075980 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.076156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.076075 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x75sc\" (UniqueName: \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.076156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.076109 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lmc\" (UniqueName: \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.076906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.076878 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.099993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.099958 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lmc\" (UniqueName: \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\") pod \"payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.155646 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.155601 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:53.177699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.177612 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.177824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.177747 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x75sc\" (UniqueName: \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.178569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.178544 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.202734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.202696 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x75sc\" (UniqueName: \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\") pod \"payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.340734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.340695 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:53.465270 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:53.465236 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce487fc_aaea_455b_9fe2_109bda57ec8b.slice/crio-8d082ea8a32ac8dd4fa99ccf70639d24ff1c37bad6925db89acad5ec3dcfe5ef WatchSource:0}: Error finding container 8d082ea8a32ac8dd4fa99ccf70639d24ff1c37bad6925db89acad5ec3dcfe5ef: Status 404 returned error can't find the container with id 8d082ea8a32ac8dd4fa99ccf70639d24ff1c37bad6925db89acad5ec3dcfe5ef Oct 30 07:35:53.466942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.466913 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 07:35:53.645755 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:35:53.645720 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffde2fa_572a_473d_95d7_c9cf252dfb42.slice/crio-cc9d3bc525cce9ad28a99cfe6f68051eb892a65fd3ab67904ae7363225c0d657 WatchSource:0}: Error finding container cc9d3bc525cce9ad28a99cfe6f68051eb892a65fd3ab67904ae7363225c0d657: Status 404 returned error can't find the container with id cc9d3bc525cce9ad28a99cfe6f68051eb892a65fd3ab67904ae7363225c0d657 Oct 30 07:35:53.646129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:53.646109 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 07:35:54.387214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.387177 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" event={"ID":"dce487fc-aaea-455b-9fe2-109bda57ec8b","Type":"ContainerStarted","Data":"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386"} Oct 30 07:35:54.387758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.387238 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" event={"ID":"dce487fc-aaea-455b-9fe2-109bda57ec8b","Type":"ContainerStarted","Data":"8d082ea8a32ac8dd4fa99ccf70639d24ff1c37bad6925db89acad5ec3dcfe5ef"} Oct 30 07:35:54.387758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.387360 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:35:54.388957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.388930 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" event={"ID":"3ffde2fa-572a-473d-95d7-c9cf252dfb42","Type":"ContainerStarted","Data":"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38"} Oct 30 07:35:54.389058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.388972 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" event={"ID":"3ffde2fa-572a-473d-95d7-c9cf252dfb42","Type":"ContainerStarted","Data":"cc9d3bc525cce9ad28a99cfe6f68051eb892a65fd3ab67904ae7363225c0d657"} Oct 30 07:35:54.389058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.388994 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:35:54.429855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.429804 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" podStartSLOduration=2.429785742 podStartE2EDuration="2.429785742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:54.427848212 +0000 UTC m=+2318.118765678" watchObservedRunningTime="2026-10-30 07:35:54.429785742 +0000 UTC m=+2318.120703169" Oct 30 07:35:54.472700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.472604 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" podStartSLOduration=2.472583075 podStartE2EDuration="2.472583075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:35:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:35:54.469599501 +0000 UTC m=+2318.160516949" watchObservedRunningTime="2026-10-30 07:35:54.472583075 +0000 UTC m=+2318.163500514" Oct 30 07:35:54.529705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.529670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:54.654298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:54.654217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:56.261194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:56.261153 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:57.457557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.457508 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" podUID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" containerName="main" containerID="cri-o://398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5" gracePeriod=45 Oct 30 07:35:57.747594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.747562 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:35:57.843400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843368 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\" (UniqueName: \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843418 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843440 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843473 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843520 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843737 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\") pod \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\" (UID: \"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96\") " Oct 30 07:35:57.843818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843759 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:57.843818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843776 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:57.843929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.843873 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:57.844381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.844354 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:57.844381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.844381 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:57.844550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.844394 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:57.845807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.845789 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:35:57.845885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.845819 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "kube-api-access-pjsnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:35:57.845950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.845918 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs" pod "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" (UID: "f7ff55f4-3b03-42f2-a21d-0f0c0221ea96"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:35:57.865599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.865568 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:57.946126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.946089 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:57.946126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.946126 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjsnz\" (UniqueName: \"kubernetes.io/projected/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-kube-api-access-pjsnz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:57.946348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:57.946140 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:35:58.420693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.420626 5894 generic.go:356] "Generic (PLEG): container finished" podID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" containerID="398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5" exitCode=0 Oct 30 07:35:58.420861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.420705 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" Oct 30 07:35:58.420861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.420665 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" event={"ID":"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96","Type":"ContainerDied","Data":"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5"} Oct 30 07:35:58.420861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.420794 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt" event={"ID":"f7ff55f4-3b03-42f2-a21d-0f0c0221ea96","Type":"ContainerDied","Data":"2a7b98d7556c81b680ddc7fa8e3a0de6acf0e87901349814bf8ac12fe77f5ebc"} Oct 30 07:35:58.420861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.420812 5894 scope.go:122] "RemoveContainer" containerID="398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5" Oct 30 07:35:58.433851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.433803 5894 scope.go:122] "RemoveContainer" containerID="398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5" Oct 30 07:35:58.434210 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:35:58.434185 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5\": container with ID starting with 398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5 not found: ID does not exist" containerID="398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5" Oct 30 07:35:58.434356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.434222 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5"} err="failed to get container status \"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5\": rpc error: code = NotFound desc = could not find container \"398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5\": container with ID starting with 398d9509e793eb2af14e7a4b1ca8889c80e3ea5e26dc3088f1f02fe70784aac5 not found: ID does not exist" Oct 30 07:35:58.476516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.476483 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:35:58.488614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:58.488583 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bd38cf/auth-test-model-bd38cf-kserve-65f4bdfcd9-54vjt"] Oct 30 07:35:59.606409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:59.606368 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:35:59.728482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:35:59.728371 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:00.432199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:00.432160 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" path="/var/lib/kubelet/pods/f7ff55f4-3b03-42f2-a21d-0f0c0221ea96/volumes" Oct 30 07:36:01.396458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:01.396423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:02.412629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:02.412595 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:36:02.413107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:02.412801 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" containerID="cri-o://7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e" gracePeriod=30 Oct 30 07:36:03.112492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:03.112454 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:04.831563 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:04.831526 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:36:04.855274 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:04.855239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:05.399621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:05.399592 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 07:36:05.399843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:05.399704 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 07:36:06.532841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:06.532803 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:07.737316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.737291 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:36:07.780323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780294 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780377 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780419 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780506 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780557 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780575 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780602 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780693 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780770 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\") pod \"d724f182-c79e-4a68-951e-a09891b13f4c\" (UID: \"d724f182-c79e-4a68-951e-a09891b13f4c\") " Oct 30 07:36:07.780925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780793 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:07.780925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780852 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:07.781072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.780963 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:07.781144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781072 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:07.781144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781034 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:07.781390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781367 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.781465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781395 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d724f182-c79e-4a68-951e-a09891b13f4c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.781465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781412 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.781465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781426 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.781465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.781439 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.782844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.782823 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:36:07.782955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.782834 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:07.782955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.782826 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "kube-api-access-6kzm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:07.783026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.782963 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token" pod "d724f182-c79e-4a68-951e-a09891b13f4c" (UID: "d724f182-c79e-4a68-951e-a09891b13f4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:07.882728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.882686 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d724f182-c79e-4a68-951e-a09891b13f4c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.882728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.882720 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d724f182-c79e-4a68-951e-a09891b13f4c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.882728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.882731 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:07.882984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:07.882743 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kzm4\" (UniqueName: \"kubernetes.io/projected/d724f182-c79e-4a68-951e-a09891b13f4c-kube-api-access-6kzm4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:08.327938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.327901 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:08.498943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.498910 5894 generic.go:356] "Generic (PLEG): container finished" podID="d724f182-c79e-4a68-951e-a09891b13f4c" containerID="7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e" exitCode=0 Oct 30 07:36:08.499119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.498967 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" Oct 30 07:36:08.499119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.498991 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" event={"ID":"d724f182-c79e-4a68-951e-a09891b13f4c","Type":"ContainerDied","Data":"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e"} Oct 30 07:36:08.499119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.499044 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s" event={"ID":"d724f182-c79e-4a68-951e-a09891b13f4c","Type":"ContainerDied","Data":"bb84fb9251032fe53f0cdcb4c24d1450d7594cac67297c8fbf361526281277a4"} Oct 30 07:36:08.499119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.499071 5894 scope.go:122] "RemoveContainer" containerID="7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e" Oct 30 07:36:08.509196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.509179 5894 scope.go:122] "RemoveContainer" containerID="7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e" Oct 30 07:36:08.509470 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:08.509451 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e\": container with ID starting with 7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e not found: ID does not exist" containerID="7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e" Oct 30 07:36:08.509521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.509479 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e"} err="failed to get container status \"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e\": rpc error: code = NotFound desc = could not find container \"7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e\": container with ID starting with 7fe4ca85fe0d4fe7a58c838035915c720a4b60285856e1e3c29e9ede1c6dc17e not found: ID does not exist" Oct 30 07:36:08.565585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.565553 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:36:08.578455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:08.578375 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-56h5s"] Oct 30 07:36:10.096396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:10.096360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:10.432061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:10.431982 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" path="/var/lib/kubelet/pods/d724f182-c79e-4a68-951e-a09891b13f4c/volumes" Oct 30 07:36:11.465370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.465332 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7596-7a5451"] Oct 30 07:36:11.465927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.465911 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" Oct 30 07:36:11.465927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.465926 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" Oct 30 07:36:11.466067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.465956 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" containerName="main" Oct 30 07:36:11.466067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.465961 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" containerName="main" Oct 30 07:36:11.466067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.466051 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d724f182-c79e-4a68-951e-a09891b13f4c" containerName="istio-proxy" Oct 30 07:36:11.466067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.466066 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7ff55f4-3b03-42f2-a21d-0f0c0221ea96" containerName="main" Oct 30 07:36:11.469299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.469273 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:11.488671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.488596 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-7a5451"] Oct 30 07:36:11.497135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.497095 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5z8jp\"" Oct 30 07:36:11.531875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.531837 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsfs\" (UniqueName: \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\") pod \"test-curl-7596-7a5451\" (UID: \"b7473208-a9b4-446b-958b-3b5bb5594e0c\") " pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:11.633116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.633072 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsfs\" (UniqueName: \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\") pod \"test-curl-7596-7a5451\" (UID: \"b7473208-a9b4-446b-958b-3b5bb5594e0c\") " pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:11.656986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.656955 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsfs\" (UniqueName: \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\") pod \"test-curl-7596-7a5451\" (UID: \"b7473208-a9b4-446b-958b-3b5bb5594e0c\") " pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:11.788342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.788311 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:11.969549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:11.969514 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:12.103885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:12.103862 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7596-7a5451"] Oct 30 07:36:12.527871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:12.527836 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-7a5451" event={"ID":"b7473208-a9b4-446b-958b-3b5bb5594e0c","Type":"ContainerStarted","Data":"8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b"} Oct 30 07:36:12.528273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:12.527883 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7596-7a5451" event={"ID":"b7473208-a9b4-446b-958b-3b5bb5594e0c","Type":"ContainerStarted","Data":"a65e78f8a29e638207beb9dcddd3da5b44ae9fa4381361115c267b43e89d91b0"} Oct 30 07:36:12.568147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:12.568092 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7596-7a5451" podStartSLOduration=1.5680768189999998 podStartE2EDuration="1.568076819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:12.566465888 +0000 UTC m=+2336.257383325" watchObservedRunningTime="2026-10-30 07:36:12.568076819 +0000 UTC m=+2336.258994256" Oct 30 07:36:13.541022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.540986 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7596-7a5451"] Oct 30 07:36:13.547749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.547712 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7596-7a5451"] Oct 30 07:36:13.569826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.569790 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:13.765843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.765807 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw"] Oct 30 07:36:13.766376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.766363 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" containerName="test-curl-7596-7a5451" Oct 30 07:36:13.766432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.766377 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" containerName="test-curl-7596-7a5451" Oct 30 07:36:13.766518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.766508 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" containerName="test-curl-7596-7a5451" Oct 30 07:36:13.769977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.769954 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.786849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.786821 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:36:13.790003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.789975 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw"] Oct 30 07:36:13.800715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.799124 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:36:13.800715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.799165 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:36:13.800715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.799204 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:36:13.863338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.863300 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.863547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.863498 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8694\" (UniqueName: \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.863617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.863565 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.965211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.965173 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.965397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.965266 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.965397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.965363 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8694\" (UniqueName: \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.966027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.966005 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.967622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.967606 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:13.988965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:13.988933 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8694\" (UniqueName: \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\") pod \"maas-api-revoke-sub-a6b9ad7c-v6lbw\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:14.083369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:14.083287 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:14.386231 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:14.386194 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0afa885_a2aa_495e_b874_a28218e75e82.slice/crio-044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9 WatchSource:0}: Error finding container 044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9: Status 404 returned error can't find the container with id 044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9 Oct 30 07:36:14.386983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:14.386956 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw"] Oct 30 07:36:14.541816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:14.541779 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" event={"ID":"e0afa885-a2aa-495e-b874-a28218e75e82","Type":"ContainerStarted","Data":"044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9"} Oct 30 07:36:14.542252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:14.541875 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7596-7a5451" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" containerName="test-curl-7596-7a5451" containerID="cri-o://8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b" gracePeriod=2 Oct 30 07:36:15.238017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:15.237983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:15.551057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:15.551025 5894 generic.go:356] "Generic (PLEG): container finished" podID="e0afa885-a2aa-495e-b874-a28218e75e82" containerID="99ecaf77acfeae7e2f8d4ac0ca86ca90870d2c668170d198cb20f1cac2457b5f" exitCode=0 Oct 30 07:36:15.551556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:15.551065 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" event={"ID":"e0afa885-a2aa-495e-b874-a28218e75e82","Type":"ContainerDied","Data":"99ecaf77acfeae7e2f8d4ac0ca86ca90870d2c668170d198cb20f1cac2457b5f"} Oct 30 07:36:15.584209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:15.584177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:16.772817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.772787 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:16.776391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.776370 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:16.904855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.904775 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\" (UniqueName: \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\") pod \"b7473208-a9b4-446b-958b-3b5bb5594e0c\" (UID: \"b7473208-a9b4-446b-958b-3b5bb5594e0c\") " Oct 30 07:36:16.905032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.904866 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\" (UniqueName: \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\") pod \"e0afa885-a2aa-495e-b874-a28218e75e82\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " Oct 30 07:36:16.905032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.905015 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\") pod \"e0afa885-a2aa-495e-b874-a28218e75e82\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " Oct 30 07:36:16.905168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.905083 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\") pod \"e0afa885-a2aa-495e-b874-a28218e75e82\" (UID: \"e0afa885-a2aa-495e-b874-a28218e75e82\") " Oct 30 07:36:16.905451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.905421 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca" pod "e0afa885-a2aa-495e-b874-a28218e75e82" (UID: "e0afa885-a2aa-495e-b874-a28218e75e82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:16.905652 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.905614 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:16.907184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.907150 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca" pod "e0afa885-a2aa-495e-b874-a28218e75e82" (UID: "e0afa885-a2aa-495e-b874-a28218e75e82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:36:16.907331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.907305 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs" pod "b7473208-a9b4-446b-958b-3b5bb5594e0c" (UID: "b7473208-a9b4-446b-958b-3b5bb5594e0c"). InnerVolumeSpecName "kube-api-access-njsfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:16.907396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.907323 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694" pod "e0afa885-a2aa-495e-b874-a28218e75e82" (UID: "e0afa885-a2aa-495e-b874-a28218e75e82"). InnerVolumeSpecName "kube-api-access-s8694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:16.923106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:16.923074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:17.007414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.007384 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8694\" (UniqueName: \"kubernetes.io/projected/e0afa885-a2aa-495e-b874-a28218e75e82-kube-api-access-s8694\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:17.007414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.007411 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0afa885-a2aa-495e-b874-a28218e75e82-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:17.007674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.007424 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njsfs\" (UniqueName: \"kubernetes.io/projected/b7473208-a9b4-446b-958b-3b5bb5594e0c-kube-api-access-njsfs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:17.041077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.041042 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:17.357555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.357478 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 07:36:17.358713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.358409 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e0afa885-a2aa-495e-b874-a28218e75e82" containerName="revoke-keys" Oct 30 07:36:17.358713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.358435 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0afa885-a2aa-495e-b874-a28218e75e82" containerName="revoke-keys" Oct 30 07:36:17.358713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.358659 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0afa885-a2aa-495e-b874-a28218e75e82" containerName="revoke-keys" Oct 30 07:36:17.363595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.363564 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.382577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.382549 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 07:36:17.390656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.388242 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e193ee-openshift-default-dockercfg-66gfh\"" Oct 30 07:36:17.412462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412432 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412485 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412698 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412749 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412786 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnsjv\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412815 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.412939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.412921 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.413017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.413005 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.413163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.413146 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.431180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.430958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:17.514496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514462 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.514751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514509 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.514751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514552 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.514751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514597 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.514751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514699 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514756 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnsjv\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514805 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514838 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514919 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.514985 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.515007 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.515062 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.515495 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.515831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.515807 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.517410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.517385 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.517874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.517852 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.539182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.539144 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnsjv\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.544124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.544090 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\") pod \"e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.565694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.563889 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:17.617213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.617124 5894 generic.go:356] "Generic (PLEG): container finished" podID="b7473208-a9b4-446b-958b-3b5bb5594e0c" containerID="8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b" exitCode=137 Oct 30 07:36:17.617676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.617618 5894 scope.go:122] "RemoveContainer" containerID="8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b" Oct 30 07:36:17.617996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.617980 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:17.626978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.626928 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" event={"ID":"e0afa885-a2aa-495e-b874-a28218e75e82","Type":"ContainerDied","Data":"044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9"} Oct 30 07:36:17.627282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.627249 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="044dc746af226a365188a1f3f1acf64471aafd1cb1fec00bc720038c7b15c8d9" Oct 30 07:36:17.627503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.627488 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw" Oct 30 07:36:17.683825 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:17.683507 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-7a5451\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:17.690830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.689837 5894 scope.go:122] "RemoveContainer" containerID="8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b" Oct 30 07:36:17.690830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.690788 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:17.691588 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:17.691516 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b\": container with ID starting with 8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b not found: ID does not exist" containerID="8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b" Oct 30 07:36:17.691588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:17.691557 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b"} err="failed to get container status \"8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b\": rpc error: code = NotFound desc = could not find container \"8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b\": container with ID starting with 8bc9c7370eede5c8d97848b4a8ff1a5e01aacd8e612ebeddb715cdc93bfd208b not found: ID does not exist" Oct 30 07:36:17.699900 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:17.699863 5894 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7596-7a5451\" is forbidden: User \"system:node:ip-10-0-143-90.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-90.ec2.internal' and this object" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" pod="opendatahub/test-curl-7596-7a5451" Oct 30 07:36:18.054468 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:18.054426 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bb61db_2d23_4aac_9ade_184a5e5ed99f.slice/crio-33a15e3be9527ed2266d09ef40dd9cff0a5430a5099b33cec2eba2e387a31188 WatchSource:0}: Error finding container 33a15e3be9527ed2266d09ef40dd9cff0a5430a5099b33cec2eba2e387a31188: Status 404 returned error can't find the container with id 33a15e3be9527ed2266d09ef40dd9cff0a5430a5099b33cec2eba2e387a31188 Oct 30 07:36:18.055320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.055244 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 07:36:18.057746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.057708 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.057886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.057778 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.057886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.057805 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:36:18.433202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.433123 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7473208-a9b4-446b-958b-3b5bb5594e0c" path="/var/lib/kubelet/pods/b7473208-a9b4-446b-958b-3b5bb5594e0c/volumes" Oct 30 07:36:18.638516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.638479 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerStarted","Data":"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910"} Oct 30 07:36:18.638775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.638536 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerStarted","Data":"33a15e3be9527ed2266d09ef40dd9cff0a5430a5099b33cec2eba2e387a31188"} Oct 30 07:36:18.688332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.688221 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podStartSLOduration=1.68820778 podStartE2EDuration="1.68820778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:18.686351189 +0000 UTC m=+2342.377268626" watchObservedRunningTime="2026-10-30 07:36:18.68820778 +0000 UTC m=+2342.379125217" Oct 30 07:36:18.691427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.691404 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:18.728499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.728464 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:18.901375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.901341 5894 scope.go:122] "RemoveContainer" containerID="fa7ac175d3d7dc6e89fca35d97bf32cb1eb65cbc7ca6972b8b1b4f126a826682" Oct 30 07:36:18.926178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.926151 5894 scope.go:122] "RemoveContainer" containerID="036bf5b0cf82b9753eb05b835591a1eae48ccfb1482de4b1c950857ff2d1cb13" Oct 30 07:36:18.936785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:18.936764 5894 scope.go:122] "RemoveContainer" containerID="25dcf9ccacd73c0065dd3b1cf85c97ed6cd5c397348450a867b07700c2072fc8" Oct 30 07:36:19.697419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:19.697383 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:20.444022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:20.443986 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:20.651343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:20.651311 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:20.652238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:20.652215 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:36:21.039473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:21.039437 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:21.168400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:21.168354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:22.314005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:22.313972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:22.461262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:22.461219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:22.582548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:22.582460 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:23.757658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:23.757613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:23.878958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:23.878922 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:23.991476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:23.991431 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 07:36:23.997948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:23.997919 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 07:36:23.998113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:23.998028 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.002452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.002429 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.018754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.018727 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-e193ee-dockercfg-2gk5k\"" Oct 30 07:36:24.022820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.022784 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e193ee\"" Oct 30 07:36:24.023153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.023131 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 07:36:24.034889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.034853 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 07:36:24.107716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.107678 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vrb\" (UniqueName: \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.107911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.107751 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.107911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.107871 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.108000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.107986 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8jq\" (UniqueName: \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.209246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.209201 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8jq\" (UniqueName: \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.209489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.209350 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vrb\" (UniqueName: \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.209489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.209405 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.209734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.209530 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.210825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.210800 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.212220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.212191 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.235559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.235525 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8jq\" (UniqueName: \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\") pod \"payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.237317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.237295 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vrb\" (UniqueName: \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\") pod \"payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.249674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.249618 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc"] Oct 30 07:36:24.261233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.261201 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.274276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.274242 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc"] Oct 30 07:36:24.277305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.277277 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:36:24.277443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.277317 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:36:24.277443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.277333 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:36:24.277443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.277354 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:36:24.315943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.315909 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:24.325367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.325338 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:24.413399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.413148 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.413399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.413223 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tftjb\" (UniqueName: \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.413399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.413297 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.480578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.480486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:24.519268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.518085 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.519268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.518192 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tftjb\" (UniqueName: \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.519268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.518401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.521195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.521132 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.525359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.525336 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.545147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.545112 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tftjb\" (UniqueName: \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\") pod \"maas-api-revoke-sub-086f927b-x7ltc\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.579271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.579234 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:24.632061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.632031 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 07:36:24.633154 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:24.633127 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b9d412_dc54_44fd_b60f_fab3b7b2b2fb.slice/crio-0bb04e88843b45de6111b1550cde3b9cd3287da348a1eebc4a89c31c4d200985 WatchSource:0}: Error finding container 0bb04e88843b45de6111b1550cde3b9cd3287da348a1eebc4a89c31c4d200985: Status 404 returned error can't find the container with id 0bb04e88843b45de6111b1550cde3b9cd3287da348a1eebc4a89c31c4d200985 Oct 30 07:36:24.647653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.647574 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 07:36:24.686168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.686135 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" event={"ID":"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb","Type":"ContainerStarted","Data":"0bb04e88843b45de6111b1550cde3b9cd3287da348a1eebc4a89c31c4d200985"} Oct 30 07:36:24.689260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.689231 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" event={"ID":"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e","Type":"ContainerStarted","Data":"5110347951ff9e6375adce95b2f582da39a343dd69723d78aebdb34f392e8ebd"} Oct 30 07:36:24.891750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:24.891723 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc"] Oct 30 07:36:24.892190 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:24.891848 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8491a39_dda0_487b_b2d6_6833422828ec.slice/crio-0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93 WatchSource:0}: Error finding container 0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93: Status 404 returned error can't find the container with id 0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93 Oct 30 07:36:25.697619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.697583 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" event={"ID":"c8491a39-dda0-487b-b2d6-6833422828ec","Type":"ContainerStarted","Data":"0fd71da603130d2af3f2e3e57492d6e348a28b1a78d342eea8b221154300bc8b"} Oct 30 07:36:25.697874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.697671 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" event={"ID":"c8491a39-dda0-487b-b2d6-6833422828ec","Type":"ContainerStarted","Data":"0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93"} Oct 30 07:36:25.699360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.699330 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" event={"ID":"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb","Type":"ContainerStarted","Data":"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781"} Oct 30 07:36:25.699559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.699411 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:25.701099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.701075 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" event={"ID":"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e","Type":"ContainerStarted","Data":"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d"} Oct 30 07:36:25.701245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.701123 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:25.754227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.754156 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" podStartSLOduration=1.7541416600000002 podStartE2EDuration="1.75414166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:25.73951511 +0000 UTC m=+2349.430432549" watchObservedRunningTime="2026-10-30 07:36:25.75414166 +0000 UTC m=+2349.445059097" Oct 30 07:36:25.785386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.785339 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" podStartSLOduration=2.785327219 podStartE2EDuration="2.785327219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:25.784216524 +0000 UTC m=+2349.475133961" watchObservedRunningTime="2026-10-30 07:36:25.785327219 +0000 UTC m=+2349.476244655" Oct 30 07:36:25.834705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:25.834623 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" podStartSLOduration=2.834610272 podStartE2EDuration="2.834610272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:25.831233644 +0000 UTC m=+2349.522151083" watchObservedRunningTime="2026-10-30 07:36:25.834610272 +0000 UTC m=+2349.525527700" Oct 30 07:36:26.182835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:26.182795 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:26.709647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:26.709597 5894 generic.go:356] "Generic (PLEG): container finished" podID="c8491a39-dda0-487b-b2d6-6833422828ec" containerID="0fd71da603130d2af3f2e3e57492d6e348a28b1a78d342eea8b221154300bc8b" exitCode=0 Oct 30 07:36:26.709804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:26.709675 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" event={"ID":"c8491a39-dda0-487b-b2d6-6833422828ec","Type":"ContainerDied","Data":"0fd71da603130d2af3f2e3e57492d6e348a28b1a78d342eea8b221154300bc8b"} Oct 30 07:36:27.811024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:27.810987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:27.888607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:27.888575 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:28.066270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.066243 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\" (UniqueName: \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\") pod \"c8491a39-dda0-487b-b2d6-6833422828ec\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " Oct 30 07:36:28.066471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.066330 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\") pod \"c8491a39-dda0-487b-b2d6-6833422828ec\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " Oct 30 07:36:28.066471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.066457 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\") pod \"c8491a39-dda0-487b-b2d6-6833422828ec\" (UID: \"c8491a39-dda0-487b-b2d6-6833422828ec\") " Oct 30 07:36:28.066745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.066721 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca" pod "c8491a39-dda0-487b-b2d6-6833422828ec" (UID: "c8491a39-dda0-487b-b2d6-6833422828ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:36:28.066927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.066907 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:28.068425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.068404 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb" pod "c8491a39-dda0-487b-b2d6-6833422828ec" (UID: "c8491a39-dda0-487b-b2d6-6833422828ec"). InnerVolumeSpecName "kube-api-access-tftjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:28.068565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.068546 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca" pod "c8491a39-dda0-487b-b2d6-6833422828ec" (UID: "c8491a39-dda0-487b-b2d6-6833422828ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:36:28.170146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.170112 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8491a39-dda0-487b-b2d6-6833422828ec-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:28.170146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.170145 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tftjb\" (UniqueName: \"kubernetes.io/projected/c8491a39-dda0-487b-b2d6-6833422828ec-kube-api-access-tftjb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:28.724543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.724438 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" event={"ID":"c8491a39-dda0-487b-b2d6-6833422828ec","Type":"ContainerDied","Data":"0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93"} Oct 30 07:36:28.724543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.724513 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0579f257ac818cda23533023a9743e39556cd76064e03a7f90e331825a03cc93" Oct 30 07:36:28.724543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:28.724456 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc" Oct 30 07:36:29.541049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:29.541018 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:29.662367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:29.662331 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:31.330862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:31.330828 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:33.025576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:33.025537 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:33.183864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:33.183831 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:34.828993 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:34.828948 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:36:34.840754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:34.840716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:35.934996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:35.934957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:36.061656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.060153 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:36.553171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.553138 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:36:36.554486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.554432 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:36.683669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.676133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:36.721847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.721815 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 07:36:36.727867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.727839 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 07:36:36.825874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:36.820903 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:38.107345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:38.107302 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:38.354503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:38.354470 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:38.669124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:38.669035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:38.819051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:38.818989 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:39.600107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:39.600069 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:39.732765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:39.732731 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:40.359892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:40.359854 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:40.610106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:40.596220 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:36:41.379734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.379702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:41.503811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.503777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:41.525838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.525805 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 07:36:41.526436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.526422 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c8491a39-dda0-487b-b2d6-6833422828ec" containerName="revoke-keys" Oct 30 07:36:41.526489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.526438 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8491a39-dda0-487b-b2d6-6833422828ec" containerName="revoke-keys" Oct 30 07:36:41.526537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.526528 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8491a39-dda0-487b-b2d6-6833422828ec" containerName="revoke-keys" Oct 30 07:36:41.530134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.530114 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.547006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.546979 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-a91a83\"" Oct 30 07:36:41.559572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.559531 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-a91a83-dockercfg-cn6kn\"" Oct 30 07:36:41.562974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.562942 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 07:36:41.590867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.590829 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4fhp\" (UniqueName: \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.591069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.590977 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.623730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.623694 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 07:36:41.634538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.634459 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.672276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.672241 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 07:36:41.698545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.698510 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4fhp\" (UniqueName: \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.699484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.699455 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.716083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.715993 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.716547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.715904 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.716779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.716592 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79v7\" (UniqueName: \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.732029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.732000 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4fhp\" (UniqueName: \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\") pod \"payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.818539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.818503 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.818773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.818596 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79v7\" (UniqueName: \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.819389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.819367 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.846941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.846906 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:41.897656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.889388 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79v7\" (UniqueName: \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\") pod \"payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:41.980154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:41.979900 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:42.203674 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:42.203605 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0984ff_509b_4910_8f46_437583c2f376.slice/crio-b691fa3eeac657aff5d563279c993f18459a865dcc6ef42902c5d2006eba1527 WatchSource:0}: Error finding container b691fa3eeac657aff5d563279c993f18459a865dcc6ef42902c5d2006eba1527: Status 404 returned error can't find the container with id b691fa3eeac657aff5d563279c993f18459a865dcc6ef42902c5d2006eba1527 Oct 30 07:36:42.204103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.204077 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 07:36:42.312472 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:42.312439 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d9383d_2370_4553_bfb4_298acddff9d3.slice/crio-d13b9940d1ed38d701d8b8d67bf4f4eee9dbc5db0aa886bf9eb13c049e564bed WatchSource:0}: Error finding container d13b9940d1ed38d701d8b8d67bf4f4eee9dbc5db0aa886bf9eb13c049e564bed: Status 404 returned error can't find the container with id d13b9940d1ed38d701d8b8d67bf4f4eee9dbc5db0aa886bf9eb13c049e564bed Oct 30 07:36:42.312835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.312802 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 07:36:42.515898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.515813 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:42.719010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.718967 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:42.830704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.830676 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:42.892334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.892298 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" event={"ID":"4c0984ff-509b-4910-8f46-437583c2f376","Type":"ContainerStarted","Data":"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b"} Oct 30 07:36:42.892535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.892361 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" event={"ID":"4c0984ff-509b-4910-8f46-437583c2f376","Type":"ContainerStarted","Data":"b691fa3eeac657aff5d563279c993f18459a865dcc6ef42902c5d2006eba1527"} Oct 30 07:36:42.892535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.892389 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:42.894126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.894092 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" event={"ID":"d3d9383d-2370-4553-bfb4-298acddff9d3","Type":"ContainerStarted","Data":"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2"} Oct 30 07:36:42.894284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.894155 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" event={"ID":"d3d9383d-2370-4553-bfb4-298acddff9d3","Type":"ContainerStarted","Data":"d13b9940d1ed38d701d8b8d67bf4f4eee9dbc5db0aa886bf9eb13c049e564bed"} Oct 30 07:36:42.894284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.894268 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:42.935901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.935845 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" podStartSLOduration=1.93582542 podStartE2EDuration="1.93582542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:42.933587763 +0000 UTC m=+2366.624505202" watchObservedRunningTime="2026-10-30 07:36:42.93582542 +0000 UTC m=+2366.626742862" Oct 30 07:36:42.978449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:42.978397 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" podStartSLOduration=1.978381452 podStartE2EDuration="1.978381452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:42.976603296 +0000 UTC m=+2366.667520734" watchObservedRunningTime="2026-10-30 07:36:42.978381452 +0000 UTC m=+2366.669298904" Oct 30 07:36:43.610019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:43.609980 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:36:43.669462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:43.669427 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:44.276374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:44.276338 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:44.509266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:44.509227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:44.872132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:44.872096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:45.308117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.308084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:45.637501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.637403 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:45.767047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.767007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:45.814355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.814322 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 07:36:45.818845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.818822 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:45.836561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.836528 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bffe43\"" Oct 30 07:36:45.855426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.855386 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bffe43-dockercfg-b8sx5\"" Oct 30 07:36:45.860582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.860552 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 07:36:46.007927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.966959 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 07:36:46.007927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.975855 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.007927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:45.985597 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 07:36:46.007927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.005920 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.036744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.036347 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rll\" (UniqueName: \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.137574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.137540 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.137790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.137601 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtk4w\" (UniqueName: \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.137850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.137809 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.137903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.137866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rll\" (UniqueName: \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.138585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.138560 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.167689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.163615 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rll\" (UniqueName: \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\") pod \"payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.239310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.239272 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.239506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.239333 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtk4w\" (UniqueName: \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.240053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.240031 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.263421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.263340 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtk4w\" (UniqueName: \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\") pod \"payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.356773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.356727 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:46.439915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.439667 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:46.455063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.455031 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:46.673318 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:46.673278 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c381e5d_6156_45da_8738_ace575699213.slice/crio-a4f8f33d0f464d9a6e6b3c3c3f9356a9371157bbc31b6f4dc54e40a20baa9226 WatchSource:0}: Error finding container a4f8f33d0f464d9a6e6b3c3c3f9356a9371157bbc31b6f4dc54e40a20baa9226: Status 404 returned error can't find the container with id a4f8f33d0f464d9a6e6b3c3c3f9356a9371157bbc31b6f4dc54e40a20baa9226 Oct 30 07:36:46.675445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.675417 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 07:36:46.733131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.733092 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:46.796193 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:36:46.796118 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa177d1a_ab03_43f9_acb4_043698ffb066.slice/crio-2b10f3ac52473f7dbb7e4b27f2e0f6a27e5ce7bf7d58511fb803ca747e8766c9 WatchSource:0}: Error finding container 2b10f3ac52473f7dbb7e4b27f2e0f6a27e5ce7bf7d58511fb803ca747e8766c9: Status 404 returned error can't find the container with id 2b10f3ac52473f7dbb7e4b27f2e0f6a27e5ce7bf7d58511fb803ca747e8766c9 Oct 30 07:36:46.796359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.796335 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 07:36:46.862135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:46.862068 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:47.047937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.047845 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" event={"ID":"fa177d1a-ab03-43f9-acb4-043698ffb066","Type":"ContainerStarted","Data":"6c587bf9173acb24967658c393000c958ea3157574481a2376fecac8f0197e83"} Oct 30 07:36:47.047937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.047906 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" event={"ID":"fa177d1a-ab03-43f9-acb4-043698ffb066","Type":"ContainerStarted","Data":"2b10f3ac52473f7dbb7e4b27f2e0f6a27e5ce7bf7d58511fb803ca747e8766c9"} Oct 30 07:36:47.048158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.047945 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:47.049570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.049544 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" event={"ID":"0c381e5d-6156-45da-8738-ace575699213","Type":"ContainerStarted","Data":"b145a75886ecbc86c81aa455a04fe271b065eff0b24df9d8111339adaea6958f"} Oct 30 07:36:47.049737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.049587 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" event={"ID":"0c381e5d-6156-45da-8738-ace575699213","Type":"ContainerStarted","Data":"a4f8f33d0f464d9a6e6b3c3c3f9356a9371157bbc31b6f4dc54e40a20baa9226"} Oct 30 07:36:47.049737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.049690 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:47.055830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.055806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:47.093248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.093179 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" podStartSLOduration=2.093163909 podStartE2EDuration="2.093163909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:47.088038736 +0000 UTC m=+2370.778956173" watchObservedRunningTime="2026-10-30 07:36:47.093163909 +0000 UTC m=+2370.784081346" Oct 30 07:36:47.129751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:47.129704 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" podStartSLOduration=2.129689182 podStartE2EDuration="2.129689182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:36:47.12807206 +0000 UTC m=+2370.818989497" watchObservedRunningTime="2026-10-30 07:36:47.129689182 +0000 UTC m=+2370.820606632" Oct 30 07:36:48.319092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:48.319058 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:49.940811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:49.940775 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:51.580290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.580252 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerName="main" containerID="cri-o://4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844" gracePeriod=45 Oct 30 07:36:51.658777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.658733 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.199:8000/ready\": dial tcp 10.134.0.199:8000: connect: connection refused" Oct 30 07:36:51.725581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.725550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:51.889098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.889067 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:36:51.919974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.919939 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920015 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920094 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920163 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\" (UniqueName: \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920228 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:51.920343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920309 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:51.920397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920360 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920403 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\") pod \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\" (UID: \"6bc7b89e-44be-40ed-9258-a1651cdbf0ca\") " Oct 30 07:36:51.920896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920871 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:51.920896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.920895 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:51.921294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.921270 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:51.922729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.922700 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:36:51.923391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.923364 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:51.923472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:51.923383 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh" pod "6bc7b89e-44be-40ed-9258-a1651cdbf0ca" (UID: "6bc7b89e-44be-40ed-9258-a1651cdbf0ca"). InnerVolumeSpecName "kube-api-access-h6qbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:52.021705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.021668 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:52.021705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.021698 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6qbh\" (UniqueName: \"kubernetes.io/projected/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-kube-api-access-h6qbh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:52.021705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.021708 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:52.021957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.021717 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bc7b89e-44be-40ed-9258-a1651cdbf0ca-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:52.085883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.085539 5894 generic.go:356] "Generic (PLEG): container finished" podID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerID="4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844" exitCode=0 Oct 30 07:36:52.085883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.085586 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" event={"ID":"6bc7b89e-44be-40ed-9258-a1651cdbf0ca","Type":"ContainerDied","Data":"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844"} Oct 30 07:36:52.085883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.085612 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" Oct 30 07:36:52.085883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.085674 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q" event={"ID":"6bc7b89e-44be-40ed-9258-a1651cdbf0ca","Type":"ContainerDied","Data":"94e0aecc5888fe80cf2b004786f26a0ecaa5da99fc9389e0e1dbcc99eb737fff"} Oct 30 07:36:52.085883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.085703 5894 scope.go:122] "RemoveContainer" containerID="4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844" Oct 30 07:36:52.097655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.097623 5894 scope.go:122] "RemoveContainer" containerID="4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844" Oct 30 07:36:52.097956 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:52.097937 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844\": container with ID starting with 4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844 not found: ID does not exist" containerID="4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844" Oct 30 07:36:52.098024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.097966 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844"} err="failed to get container status \"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844\": rpc error: code = NotFound desc = could not find container \"4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844\": container with ID starting with 4f2622f7a5c60b7af8da2747bfcf275fc97cc322cc08b0eaefe595004c6a7844 not found: ID does not exist" Oct 30 07:36:52.138007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.137924 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:36:52.145488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.145456 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/auth-test-model-d89593-kserve-9fd8d4bfc-phs2q"] Oct 30 07:36:52.433588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:52.433511 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" path="/var/lib/kubelet/pods/6bc7b89e-44be-40ed-9258-a1651cdbf0ca/volumes" Oct 30 07:36:53.319206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:53.319172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:53.903860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:53.903831 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 07:36:53.904024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:53.903898 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 07:36:55.004197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:55.004158 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:55.132661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:55.126120 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:56.737302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:56.737270 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:58.060697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:58.060665 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 07:36:58.061115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:58.060746 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 07:36:58.525210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:58.525121 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:36:58.638052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:58.637989 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerName="main" containerID="cri-o://51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361" gracePeriod=45 Oct 30 07:36:58.942623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:58.942582 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:36:59.017791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.017757 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.017996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.017800 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\" (UniqueName: \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.017996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.017850 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\" (UniqueName: \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.017996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.017929 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.018188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.018044 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.018188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.018103 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\") pod \"4793fe07-dee8-45e2-bb0f-f8520908bef9\" (UID: \"4793fe07-dee8-45e2-bb0f-f8520908bef9\") " Oct 30 07:36:59.018780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.018584 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:59.018918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.018804 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:59.019061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.019027 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.019061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.019047 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:59.020246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.020225 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:36:59.020341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.020240 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "kube-api-access-vlqjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:36:59.020341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.020318 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs" pod "4793fe07-dee8-45e2-bb0f-f8520908bef9" (UID: "4793fe07-dee8-45e2-bb0f-f8520908bef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:36:59.120623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.120528 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.120623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.120574 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.120623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.120588 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlqjj\" (UniqueName: \"kubernetes.io/projected/4793fe07-dee8-45e2-bb0f-f8520908bef9-kube-api-access-vlqjj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.120623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.120603 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4793fe07-dee8-45e2-bb0f-f8520908bef9-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.120623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.120616 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4793fe07-dee8-45e2-bb0f-f8520908bef9-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:36:59.143625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.143586 5894 generic.go:356] "Generic (PLEG): container finished" podID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerID="51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361" exitCode=0 Oct 30 07:36:59.143625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.143642 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" Oct 30 07:36:59.143880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.143663 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" event={"ID":"4793fe07-dee8-45e2-bb0f-f8520908bef9","Type":"ContainerDied","Data":"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361"} Oct 30 07:36:59.143880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.143705 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" event={"ID":"4793fe07-dee8-45e2-bb0f-f8520908bef9","Type":"ContainerDied","Data":"af916426b9377f755005a16d6dc9750058fb68a922e7574a35ff20f8a3d92750"} Oct 30 07:36:59.143880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.143724 5894 scope.go:122] "RemoveContainer" containerID="51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361" Oct 30 07:36:59.155239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.155216 5894 scope.go:122] "RemoveContainer" containerID="51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361" Oct 30 07:36:59.155535 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:36:59.155511 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361\": container with ID starting with 51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361 not found: ID does not exist" containerID="51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361" Oct 30 07:36:59.155583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.155545 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361"} err="failed to get container status \"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361\": rpc error: code = NotFound desc = could not find container \"51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361\": container with ID starting with 51ffe66f14d3d88445b6299187169d4f740c0798992cf1b7bb85e392381a9361 not found: ID does not exist" Oct 30 07:36:59.199991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.199931 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:36:59.204157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.204126 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb"] Oct 30 07:36:59.907229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:36:59.907188 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-054ef1/auth-test-model-054ef1-kserve-5b6ddbc75c-rqzjb" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.200:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 07:37:00.233474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:00.233387 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:00.431816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:00.431782 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" path="/var/lib/kubelet/pods/4793fe07-dee8-45e2-bb0f-f8520908bef9/volumes" Oct 30 07:37:01.949233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:01.949200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:02.530010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:02.529949 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:03.351582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.351542 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:03.352539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352514 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerName="main" Oct 30 07:37:03.352706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352542 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerName="main" Oct 30 07:37:03.352706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352567 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerName="main" Oct 30 07:37:03.352706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352575 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerName="main" Oct 30 07:37:03.352883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352830 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4793fe07-dee8-45e2-bb0f-f8520908bef9" containerName="main" Oct 30 07:37:03.352883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.352852 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc7b89e-44be-40ed-9258-a1651cdbf0ca" containerName="main" Oct 30 07:37:03.358961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.358929 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.375854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.375827 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:03.383958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.383923 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1fb038\"/\"kube-root-ca.crt\"" Oct 30 07:37:03.384149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.383965 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-1fb038\"/\"openshift-service-ca.crt\"" Oct 30 07:37:03.401585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.401547 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:03.413422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.413392 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1fb038\"/\"default-dockercfg-hv729\"" Oct 30 07:37:03.421515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.421485 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-1fb038\"/\"noten-model-1fb038-kserve-self-signed-certs\"" Oct 30 07:37:03.486922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.486888 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.487118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.486937 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.487118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.487001 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.487215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.487187 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.487360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.487339 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4mj2\" (UniqueName: \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.487429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.487381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.588924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.588878 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589044 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4mj2\" (UniqueName: \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589074 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589187 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589216 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589241 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589394 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589499 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.589592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.589572 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.592445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.592420 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.592611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.592565 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.613231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.613152 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4mj2\" (UniqueName: \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\") pod \"noten-model-1fb038-kserve-6cc9756d4f-gftjt\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.645210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.645173 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:03.684562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.677943 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:03.963788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:03.963713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:04.058745 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:37:04.058715 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff1f4592_4d90_4c06_b848_602ed8b925a0.slice/crio-3d166bade7a15657fe906b065356b1174fabdeffcfa73df3472e033f38d87b2d WatchSource:0}: Error finding container 3d166bade7a15657fe906b065356b1174fabdeffcfa73df3472e033f38d87b2d: Status 404 returned error can't find the container with id 3d166bade7a15657fe906b065356b1174fabdeffcfa73df3472e033f38d87b2d Oct 30 07:37:04.059929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.059906 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:04.188130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.188094 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" event={"ID":"ff1f4592-4d90-4c06-b848-602ed8b925a0","Type":"ContainerStarted","Data":"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e"} Oct 30 07:37:04.188411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.188140 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" event={"ID":"ff1f4592-4d90-4c06-b848-602ed8b925a0","Type":"ContainerStarted","Data":"3d166bade7a15657fe906b065356b1174fabdeffcfa73df3472e033f38d87b2d"} Oct 30 07:37:04.235162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.235039 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" podStartSLOduration=1.235017548 podStartE2EDuration="1.235017548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:04.231300394 +0000 UTC m=+2387.922217831" watchObservedRunningTime="2026-10-30 07:37:04.235017548 +0000 UTC m=+2387.925934988" Oct 30 07:37:04.525900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.525806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:04.642665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.642052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:04.680107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:04.679677 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:04.830859 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:37:04.830819 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:37:05.417366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:05.417328 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:07.200907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:07.196288 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:08.471614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:08.471576 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:08.606956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:08.606924 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:09.183795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:09.183763 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:09.692481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:09.692447 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:09.731329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:09.731296 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:09.919742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:09.916244 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:10.227540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.227503 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:10.365448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.365412 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:10.595393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.595356 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:10.646653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.646602 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 07:37:10.654479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.654438 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.663236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.663197 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 07:37:10.668155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.668131 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.688708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.685252 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0f76b0\"" Oct 30 07:37:10.688708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.685360 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0f76b0-dockercfg-rslmk\"" Oct 30 07:37:10.724525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.724489 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p299q\" (UniqueName: \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.727481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.727448 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.728356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.728235 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.728693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.728671 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2jpk\" (UniqueName: \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.729593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.729570 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 07:37:10.749006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.747916 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 07:37:10.830859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.830778 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.830859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.830856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2jpk\" (UniqueName: \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.831531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.831009 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p299q\" (UniqueName: \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.831531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.831066 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.832191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.831969 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:10.832941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.832921 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.856240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.856171 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2jpk\" (UniqueName: \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\") pod \"payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:10.856378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:10.856257 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p299q\" (UniqueName: \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\") pod \"payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:11.068093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:11.068056 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:11.074612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:11.074579 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:11.175373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:11.175336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:11.379228 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:37:11.379198 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod225e48aa_8df4_4f7e_ad59_75448a449c6e.slice/crio-f8bb660c41bf3a7a899f427630f86a4114f246c90fa22e981328bd9b22bd930a WatchSource:0}: Error finding container f8bb660c41bf3a7a899f427630f86a4114f246c90fa22e981328bd9b22bd930a: Status 404 returned error can't find the container with id f8bb660c41bf3a7a899f427630f86a4114f246c90fa22e981328bd9b22bd930a Oct 30 07:37:11.380757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:11.380731 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 07:37:11.399416 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:37:11.399380 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda27d8357_d617_4303_8e82_78ff0d6c38fd.slice/crio-fbf17829b890228d09a158891dd27818ad934a7f71f6ce11ff0eee0bc425af8c WatchSource:0}: Error finding container fbf17829b890228d09a158891dd27818ad934a7f71f6ce11ff0eee0bc425af8c: Status 404 returned error can't find the container with id fbf17829b890228d09a158891dd27818ad934a7f71f6ce11ff0eee0bc425af8c Oct 30 07:37:11.399764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:11.399742 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 07:37:12.272469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.272428 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" event={"ID":"a27d8357-d617-4303-8e82-78ff0d6c38fd","Type":"ContainerStarted","Data":"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885"} Oct 30 07:37:12.272995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.272509 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:12.272995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.272537 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" event={"ID":"a27d8357-d617-4303-8e82-78ff0d6c38fd","Type":"ContainerStarted","Data":"fbf17829b890228d09a158891dd27818ad934a7f71f6ce11ff0eee0bc425af8c"} Oct 30 07:37:12.274263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.274236 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" event={"ID":"225e48aa-8df4-4f7e-ad59-75448a449c6e","Type":"ContainerStarted","Data":"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44"} Oct 30 07:37:12.274398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.274279 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" event={"ID":"225e48aa-8df4-4f7e-ad59-75448a449c6e","Type":"ContainerStarted","Data":"f8bb660c41bf3a7a899f427630f86a4114f246c90fa22e981328bd9b22bd930a"} Oct 30 07:37:12.274398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.274328 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:12.316583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.316528 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" podStartSLOduration=2.316514711 podStartE2EDuration="2.316514711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:12.314937449 +0000 UTC m=+2396.005854888" watchObservedRunningTime="2026-10-30 07:37:12.316514711 +0000 UTC m=+2396.007432147" Oct 30 07:37:12.359951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.359892 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" podStartSLOduration=2.359874847 podStartE2EDuration="2.359874847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:12.356101582 +0000 UTC m=+2396.047019020" watchObservedRunningTime="2026-10-30 07:37:12.359874847 +0000 UTC m=+2396.050792286" Oct 30 07:37:12.864011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:12.863974 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:14.520421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:14.520386 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:14.641887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:14.641847 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:15.422715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:15.422678 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:16.409713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:16.409670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:18.155002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:18.154960 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:19.698849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:19.698810 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:19.811848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:19.811817 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:19.924279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:19.924239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:21.532921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:21.532877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:23.139207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:23.139172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:23.284268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:23.284235 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 07:37:23.284452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:23.284306 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 07:37:24.691296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:24.691262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:24.720445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:24.720407 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" podUID="ff1f4592-4d90-4c06-b848-602ed8b925a0" containerName="main" containerID="cri-o://606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e" gracePeriod=45 Oct 30 07:37:25.006113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.006087 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:25.152682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152649 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\" (UniqueName: \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.152893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152749 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.152893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152828 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.152893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152858 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.153084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152963 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.153084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.152991 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\") pod \"ff1f4592-4d90-4c06-b848-602ed8b925a0\" (UID: \"ff1f4592-4d90-4c06-b848-602ed8b925a0\") " Oct 30 07:37:25.153193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153122 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:25.153321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153255 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:25.153424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153397 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:25.153716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153697 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.153716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153716 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.153873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.153725 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.155258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.155225 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:37:25.155374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.155330 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "kube-api-access-f4mj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:25.155530 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.155512 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm" pod "ff1f4592-4d90-4c06-b848-602ed8b925a0" (UID: "ff1f4592-4d90-4c06-b848-602ed8b925a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:25.255102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.255025 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4mj2\" (UniqueName: \"kubernetes.io/projected/ff1f4592-4d90-4c06-b848-602ed8b925a0-kube-api-access-f4mj2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.255102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.255052 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff1f4592-4d90-4c06-b848-602ed8b925a0-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.255102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.255064 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff1f4592-4d90-4c06-b848-602ed8b925a0-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:25.385626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.385593 5894 generic.go:356] "Generic (PLEG): container finished" podID="ff1f4592-4d90-4c06-b848-602ed8b925a0" containerID="606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e" exitCode=0 Oct 30 07:37:25.385843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.385674 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" Oct 30 07:37:25.385843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.385672 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" event={"ID":"ff1f4592-4d90-4c06-b848-602ed8b925a0","Type":"ContainerDied","Data":"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e"} Oct 30 07:37:25.385843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.385803 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt" event={"ID":"ff1f4592-4d90-4c06-b848-602ed8b925a0","Type":"ContainerDied","Data":"3d166bade7a15657fe906b065356b1174fabdeffcfa73df3472e033f38d87b2d"} Oct 30 07:37:25.385843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.385829 5894 scope.go:122] "RemoveContainer" containerID="606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e" Oct 30 07:37:25.397052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.397030 5894 scope.go:122] "RemoveContainer" containerID="606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e" Oct 30 07:37:25.397354 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:37:25.397336 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e\": container with ID starting with 606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e not found: ID does not exist" containerID="606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e" Oct 30 07:37:25.397438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.397361 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e"} err="failed to get container status \"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e\": rpc error: code = NotFound desc = could not find container \"606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e\": container with ID starting with 606e58ab3d2ca672533a22dbe4640086ff8016e24f38eb5c1d26bbd923c7050e not found: ID does not exist" Oct 30 07:37:25.439369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.439339 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:25.443198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:25.443175 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-1fb038/noten-model-1fb038-kserve-6cc9756d4f-gftjt"] Oct 30 07:37:26.428410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:26.428377 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:26.431741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:26.431715 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff1f4592-4d90-4c06-b848-602ed8b925a0" path="/var/lib/kubelet/pods/ff1f4592-4d90-4c06-b848-602ed8b925a0/volumes" Oct 30 07:37:27.805102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:27.805059 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:37:28.229701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:28.228525 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:29.696265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:29.696229 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:30.110702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:30.110099 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:31.828618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:31.828581 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:33.522089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:33.522055 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:34.829878 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:37:34.829835 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:37:35.355527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:35.355480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:36.692823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:36.692782 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 07:37:36.693271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:36.692864 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 07:37:37.248926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:37.248887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:37.578321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:37.578286 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:38.754494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:38.754444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:38.878726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:38.878579 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:39.009245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.009053 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:39.132890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.132857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:39.475359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.475319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:39.486502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.486473 5894 generic.go:356] "Generic (PLEG): container finished" podID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerID="2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910" exitCode=0 Oct 30 07:37:39.486712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.486543 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerDied","Data":"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910"} Oct 30 07:37:39.486997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.486964 5894 scope.go:122] "RemoveContainer" containerID="2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910" Oct 30 07:37:39.489266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.489232 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:39.489392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.489309 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:39.489392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.489346 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:39.696680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:39.695998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:40.327762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.327724 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:40.454651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.454601 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:40.509510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.509472 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerStarted","Data":"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb"} Oct 30 07:37:40.699077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.698992 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:37:40.699419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.699342 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:40.699582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:40.699562 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:41.156457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:41.156384 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:41.696110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:41.696071 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:41.696546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:41.696138 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:41.951043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:41.950964 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:42.207852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.207771 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:37:42.208419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.208405 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ff1f4592-4d90-4c06-b848-602ed8b925a0" containerName="main" Oct 30 07:37:42.208464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.208420 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff1f4592-4d90-4c06-b848-602ed8b925a0" containerName="main" Oct 30 07:37:42.208539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.208530 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff1f4592-4d90-4c06-b848-602ed8b925a0" containerName="main" Oct 30 07:37:42.210965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.210949 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.228069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.228045 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"kube-root-ca.crt\"" Oct 30 07:37:42.240501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.240478 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"openshift-service-ca.crt\"" Oct 30 07:37:42.240732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.240713 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"agc-cred-3cd6eb65d30c\"" Oct 30 07:37:42.240814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.240731 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"payload-processing-external-model-e2e--c1a7a7a6-dockercfg-gngtf\"" Oct 30 07:37:42.245830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.245802 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:37:42.306038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.306007 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"routing-overlay\"" Oct 30 07:37:42.306272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.306195 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e193ee\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e193ee\"" Oct 30 07:37:42.340990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.340949 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.341226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.341015 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.341226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.341167 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.341360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.341229 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4bh\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.444302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.444122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.444302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.444220 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4bh\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.444525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.444374 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.444525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.444434 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.446695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.445315 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.452716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.450127 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.452716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.450565 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.481609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.481530 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4bh\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\") pod \"payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.524913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.524885 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:42.696021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.695733 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:42.696021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.695792 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:42.850368 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:37:42.850334 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74935ffd_8a09_4d3b_99a5_f5f875748e9c.slice/crio-5c948f73bb2528a2af591899658ed2d07b5edcadd8083984e71e1cb2fe4713d6 WatchSource:0}: Error finding container 5c948f73bb2528a2af591899658ed2d07b5edcadd8083984e71e1cb2fe4713d6: Status 404 returned error can't find the container with id 5c948f73bb2528a2af591899658ed2d07b5edcadd8083984e71e1cb2fe4713d6 Oct 30 07:37:42.851422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:42.851393 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:37:43.020816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.020777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:43.530782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.530749 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" event={"ID":"74935ffd-8a09-4d3b-99a5-f5f875748e9c","Type":"ContainerStarted","Data":"629ef0d36aa8588b99616e98c75daa93f1370e5ad3327150d5c4cecf5190eeb9"} Oct 30 07:37:43.530978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.530805 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:43.530978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.530824 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" event={"ID":"74935ffd-8a09-4d3b-99a5-f5f875748e9c","Type":"ContainerStarted","Data":"5c948f73bb2528a2af591899658ed2d07b5edcadd8083984e71e1cb2fe4713d6"} Oct 30 07:37:43.692117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.692074 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:43.692415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:43.692380 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:44.696497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:44.696455 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:44.696975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:44.696516 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:44.752165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:44.752127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:45.692079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:45.692038 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 07:37:45.692412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:45.692379 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 07:37:46.680621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.680589 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:46.695897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.695866 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:37:46.696308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.696289 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:37:46.697017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.696998 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 07:37:46.759843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.759783 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" podStartSLOduration=4.759766111 podStartE2EDuration="4.759766111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:43.592921959 +0000 UTC m=+2427.283839397" watchObservedRunningTime="2026-10-30 07:37:46.759766111 +0000 UTC m=+2430.450683549" Oct 30 07:37:46.793964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.793931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:46.924256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:46.924220 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:47.742532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:47.742500 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:37:47.743072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:47.742777 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" containerID="cri-o://dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95" gracePeriod=30 Oct 30 07:37:48.532953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:48.532916 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:50.247878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:50.247840 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:51.768882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:51.768846 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:37:51.769290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:51.768903 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:37:51.843083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:51.843038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:52.515165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:52.515125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:53.014938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.014910 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:37:53.106289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106254 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106313 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106369 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106405 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106441 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106498 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106565 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106657 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106680 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106824 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\") pod \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\" (UID: \"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c\") " Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106877 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:53.106898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.106894 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:37:53.107241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107082 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:53.107374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107348 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:53.107554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107533 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.107672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107562 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.107672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107578 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.107672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107592 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.107672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.107607 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.109100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.109065 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:37:53.109232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.109148 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "kube-api-access-4dtcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:53.109451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.109428 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:37:53.109724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.109702 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy" pod "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" (UID: "ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:37:53.209265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.209222 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.209265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.209259 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtcd\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-kube-api-access-4dtcd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.209265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.209270 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.209517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.209280 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:37:53.518796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.518682 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:53.617740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.617700 5894 generic.go:356] "Generic (PLEG): container finished" podID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerID="dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95" exitCode=0 Oct 30 07:37:53.617929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.617753 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" Oct 30 07:37:53.617929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.617774 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" event={"ID":"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c","Type":"ContainerDied","Data":"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95"} Oct 30 07:37:53.617929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.617828 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222" event={"ID":"ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c","Type":"ContainerDied","Data":"9d9610516a5ab3142a0e326284e9cd56fb0bcf9b149c2f76a33b5f8040d987c3"} Oct 30 07:37:53.617929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.617858 5894 scope.go:122] "RemoveContainer" containerID="dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95" Oct 30 07:37:53.629270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.629246 5894 scope.go:122] "RemoveContainer" containerID="dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95" Oct 30 07:37:53.629562 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:37:53.629539 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95\": container with ID starting with dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95 not found: ID does not exist" containerID="dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95" Oct 30 07:37:53.629662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.629569 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95"} err="failed to get container status \"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95\": rpc error: code = NotFound desc = could not find container \"dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95\": container with ID starting with dc07a348a0e5dff5e8bef73aa434af24ee35a91e3efbff3c695422f4fe5dcb95 not found: ID does not exist" Oct 30 07:37:53.672292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.672258 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:37:53.684909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:53.684871 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-8x222"] Oct 30 07:37:54.102501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.102467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:54.435028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.434949 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" path="/var/lib/kubelet/pods/ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c/volumes" Oct 30 07:37:54.541502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.541468 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:37:54.581177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.581144 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 07:37:54.582205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.582182 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" Oct 30 07:37:54.582278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.582208 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" Oct 30 07:37:54.582367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.582356 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccbdc66c-1f3e-4f70-a003-8cdc6f9af59c" containerName="istio-proxy" Oct 30 07:37:54.586477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.586453 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.614221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.614178 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 07:37:54.626450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.626423 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6051b2-openshift-default-dockercfg-qqq6l\"" Oct 30 07:37:54.629600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629573 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.629781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629618 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk4xv\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.629781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629719 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.629781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629772 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.629939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629910 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.630077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.629995 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.630141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.630117 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.630223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.630154 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.630357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.630341 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734307 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734471 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734539 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734588 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734618 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734716 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734830 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734865 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk4xv\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.734995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.734909 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.735421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.735393 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.740074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.735775 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.740074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.736214 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.740074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.737695 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.740074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.738429 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.741827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.741798 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.742443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.742412 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.777657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.764415 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:54.777657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.765321 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk4xv\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.783386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.782079 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\") pod \"e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.901166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.900228 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:54.951551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:54.951519 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:55.233906 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:37:55.233878 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0f3c81_6db3_48cb_9930_3f3af3ef896e.slice/crio-3b854ea6149f9bb12be78061dacc2660837ddb2da90c66b121efcdb2c60bd935 WatchSource:0}: Error finding container 3b854ea6149f9bb12be78061dacc2660837ddb2da90c66b121efcdb2c60bd935: Status 404 returned error can't find the container with id 3b854ea6149f9bb12be78061dacc2660837ddb2da90c66b121efcdb2c60bd935 Oct 30 07:37:55.234583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.234556 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 07:37:55.236751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.236719 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:55.236828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.236800 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:55.236864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.236829 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:37:55.432106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.432058 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:55.641106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.641067 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerStarted","Data":"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9"} Oct 30 07:37:55.641106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.641125 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerStarted","Data":"3b854ea6149f9bb12be78061dacc2660837ddb2da90c66b121efcdb2c60bd935"} Oct 30 07:37:55.700427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.700364 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podStartSLOduration=1.7003436299999999 podStartE2EDuration="1.70034363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:37:55.69928496 +0000 UTC m=+2439.390202397" watchObservedRunningTime="2026-10-30 07:37:55.70034363 +0000 UTC m=+2439.391261068" Oct 30 07:37:55.901466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:55.901373 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:56.914923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:56.914845 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:57.092103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:57.092052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:57.421369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:57.421321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:57.654256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:57.654227 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:57.655177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:57.655154 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:37:58.233215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:58.233174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:58.373538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:58.373453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:58.934910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:58.934874 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:37:59.912297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:37:59.912249 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:00.015305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.015266 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 07:38:00.018702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.018679 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.040009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.039966 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6051b2\"" Oct 30 07:38:00.046709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.046676 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 07:38:00.054679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.054612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:00.059954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.059923 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 07:38:00.064145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.064121 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.093487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.093456 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6051b2-dockercfg-xrsw9\"" Oct 30 07:38:00.096224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.096194 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 07:38:00.114146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.114100 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.114146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.114150 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff79x\" (UniqueName: \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.229504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.229404 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kst\" (UniqueName: \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.229504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.229485 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.229816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.229664 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.229816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.229700 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff79x\" (UniqueName: \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.231177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.231148 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.257136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.257104 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff79x\" (UniqueName: \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\") pod \"payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.330660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.330608 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kst\" (UniqueName: \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.330852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.330678 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.331391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.331372 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.332434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.332417 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:00.355283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.355246 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kst\" (UniqueName: \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\") pod \"payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.382144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.382096 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:00.649241 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:00.649206 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc853d65c_6510_4fc7_addd_a2057dddb2bd.slice/crio-c0eb2d4aa720208ab0b1eeb739e0e5a4d100e56b6c494c2486ed44f04c262d5c WatchSource:0}: Error finding container c0eb2d4aa720208ab0b1eeb739e0e5a4d100e56b6c494c2486ed44f04c262d5c: Status 404 returned error can't find the container with id c0eb2d4aa720208ab0b1eeb739e0e5a4d100e56b6c494c2486ed44f04c262d5c Oct 30 07:38:00.649896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.649812 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 07:38:00.680558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.680526 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" event={"ID":"c853d65c-6510-4fc7-addd-a2057dddb2bd","Type":"ContainerStarted","Data":"c0eb2d4aa720208ab0b1eeb739e0e5a4d100e56b6c494c2486ed44f04c262d5c"} Oct 30 07:38:00.708281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.708244 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 07:38:00.712955 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:00.712930 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007e648c_d5b5_4b6c_9ece_748e0b79cfd0.slice/crio-a7e7a12bf6c3380bc487e3e9ab667840b5caac4f6dac4fa6a2176339ddcda902 WatchSource:0}: Error finding container a7e7a12bf6c3380bc487e3e9ab667840b5caac4f6dac4fa6a2176339ddcda902: Status 404 returned error can't find the container with id a7e7a12bf6c3380bc487e3e9ab667840b5caac4f6dac4fa6a2176339ddcda902 Oct 30 07:38:00.883473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:00.883437 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:01.400473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.400419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:01.689264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.689176 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" event={"ID":"c853d65c-6510-4fc7-addd-a2057dddb2bd","Type":"ContainerStarted","Data":"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b"} Oct 30 07:38:01.689430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.689264 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:01.690966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.690939 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" event={"ID":"007e648c-d5b5-4b6c-9ece-748e0b79cfd0","Type":"ContainerStarted","Data":"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d"} Oct 30 07:38:01.691105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.690981 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" event={"ID":"007e648c-d5b5-4b6c-9ece-748e0b79cfd0","Type":"ContainerStarted","Data":"a7e7a12bf6c3380bc487e3e9ab667840b5caac4f6dac4fa6a2176339ddcda902"} Oct 30 07:38:01.691105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.691002 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:01.732776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.732725 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" podStartSLOduration=2.732712168 podStartE2EDuration="2.732712168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:01.731909442 +0000 UTC m=+2445.422826878" watchObservedRunningTime="2026-10-30 07:38:01.732712168 +0000 UTC m=+2445.423629608" Oct 30 07:38:01.777916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:01.777848 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" podStartSLOduration=1.777829035 podStartE2EDuration="1.777829035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:01.775247246 +0000 UTC m=+2445.466164683" watchObservedRunningTime="2026-10-30 07:38:01.777829035 +0000 UTC m=+2445.468746473" Oct 30 07:38:02.095863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:02.095829 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:38:02.689910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:02.689878 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:02.829488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:02.829455 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:03.273924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:03.273884 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:03.438487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:03.438450 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:04.028065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:04.028025 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:38:04.028886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:04.028845 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" containerID="cri-o://e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1" gracePeriod=30 Oct 30 07:38:04.553013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:04.552975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:04.758728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:04.758687 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:04.856902 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:38:04.856810 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:38:05.747619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:05.747583 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:05.894383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:05.894342 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:06.060381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:06.060341 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:06.399239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:06.399143 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:06.552963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:06.552925 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:06.700081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:06.700000 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:06.823616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:06.823584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:07.199239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:07.199205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:07.337439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:07.337414 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:07.775177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:07.775146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:07.902740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:07.902700 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:08.744768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:08.744700 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:08.858518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:08.858487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:09.014860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.014781 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:38:09.018134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.018116 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.050093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.050064 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"openshift-service-ca.crt\"" Oct 30 07:38:09.050244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.050109 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"routing-overlay\"" Oct 30 07:38:09.050244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.050155 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"kube-root-ca.crt\"" Oct 30 07:38:09.050468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.050443 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bffe43\"" Oct 30 07:38:09.053342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.053320 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:38:09.068092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.068049 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.068241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.068110 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfsz\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.068241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.068183 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.068357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.068272 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.123089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.123051 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"agc-cred-7c27d1b41fae\"" Oct 30 07:38:09.123269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.123168 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"payload-processing-external-model-e2e--4ef33b46-dockercfg-zr4ng\"" Oct 30 07:38:09.169897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.169859 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.170108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.170091 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.170186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.170125 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfsz\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.170248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.170209 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.170851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.170819 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.171117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.171093 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.173095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.173063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.200678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.200571 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfsz\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.333408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.332596 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.389895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.389864 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:38:09.475193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475160 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475312 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475345 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475376 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475394 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475440 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475481 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475509 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475509 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:38:09.475860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475583 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\") pod \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\" (UID: \"ea7dd5e1-0470-46d2-8f35-4ddac39b881f\") " Oct 30 07:38:09.475927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475912 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:38:09.475991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.475937 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:38:09.476198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.476184 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.476271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.476207 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.476271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.476222 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.476785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.476698 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:38:09.476785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.476763 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:38:09.478330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.478307 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:38:09.478456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.478402 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "kube-api-access-mkbfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:38:09.478613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.478590 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:38:09.478821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.478802 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token" pod "ea7dd5e1-0470-46d2-8f35-4ddac39b881f" (UID: "ea7dd5e1-0470-46d2-8f35-4ddac39b881f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577467 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbfx\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-kube-api-access-mkbfx\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577501 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577510 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577518 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577525 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.577554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.577533 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea7dd5e1-0470-46d2-8f35-4ddac39b881f-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:38:09.656050 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:09.656022 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33622ca_6b90_46c5_904e_c4112bb4aabe.slice/crio-8f11ef6af9e3d6a4a7b94f3fb6b4436a4f5f4801ce5c3ca28fc0488236be771b WatchSource:0}: Error finding container 8f11ef6af9e3d6a4a7b94f3fb6b4436a4f5f4801ce5c3ca28fc0488236be771b: Status 404 returned error can't find the container with id 8f11ef6af9e3d6a4a7b94f3fb6b4436a4f5f4801ce5c3ca28fc0488236be771b Oct 30 07:38:09.656881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.656854 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:38:09.769916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.769883 5894 generic.go:356] "Generic (PLEG): container finished" podID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerID="e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1" exitCode=0 Oct 30 07:38:09.770109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.769942 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" Oct 30 07:38:09.770109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.769962 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" event={"ID":"ea7dd5e1-0470-46d2-8f35-4ddac39b881f","Type":"ContainerDied","Data":"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1"} Oct 30 07:38:09.770109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.770051 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d" event={"ID":"ea7dd5e1-0470-46d2-8f35-4ddac39b881f","Type":"ContainerDied","Data":"983c5a177106cb0539873ba683b22bfcaabd4bf41fa9b786a111c333c7a934c5"} Oct 30 07:38:09.770109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.770079 5894 scope.go:122] "RemoveContainer" containerID="e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1" Oct 30 07:38:09.778364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.778329 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" event={"ID":"a33622ca-6b90-46c5-904e-c4112bb4aabe","Type":"ContainerStarted","Data":"ff5b8cc1f36b0579c2c23825d307141804d7960b7487e2a0439017a0cabedf6d"} Oct 30 07:38:09.778529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.778393 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" event={"ID":"a33622ca-6b90-46c5-904e-c4112bb4aabe","Type":"ContainerStarted","Data":"8f11ef6af9e3d6a4a7b94f3fb6b4436a4f5f4801ce5c3ca28fc0488236be771b"} Oct 30 07:38:09.778529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.778421 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:09.790987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.790964 5894 scope.go:122] "RemoveContainer" containerID="e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1" Oct 30 07:38:09.791423 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:38:09.791395 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1\": container with ID starting with e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1 not found: ID does not exist" containerID="e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1" Oct 30 07:38:09.791509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.791435 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1"} err="failed to get container status \"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1\": rpc error: code = NotFound desc = could not find container \"e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1\": container with ID starting with e28ce446faaf099ecb0a9eaa4c5e5129ec82e52884d7ab66f7b383d25191bea1 not found: ID does not exist" Oct 30 07:38:09.831144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.831037 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" podStartSLOduration=1.8310235480000001 podStartE2EDuration="1.831023548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:09.827850773 +0000 UTC m=+2453.518768210" watchObservedRunningTime="2026-10-30 07:38:09.831023548 +0000 UTC m=+2453.521940984" Oct 30 07:38:09.844494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.844466 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:09.867988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.867951 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:38:09.873354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.873326 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-x876d"] Oct 30 07:38:09.965182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:09.965148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:10.002108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.002075 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 07:38:10.002792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.002775 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" Oct 30 07:38:10.002857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.002793 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" Oct 30 07:38:10.002924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.002915 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" containerName="istio-proxy" Oct 30 07:38:10.006053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.006031 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.026209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.026175 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b3a7d7-dockercfg-tlktp\"" Oct 30 07:38:10.026666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.026215 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b3a7d7\"" Oct 30 07:38:10.029248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.029220 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 07:38:10.085147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.085069 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.085308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.085143 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq56f\" (UniqueName: \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.111822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.111794 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 07:38:10.128657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.120471 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.158755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.149438 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 07:38:10.188263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.188225 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.188500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.188287 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq56f\" (UniqueName: \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.188500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.188354 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.188500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.188381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rntq\" (UniqueName: \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.189388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.189361 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.214729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.214698 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq56f\" (UniqueName: \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\") pod \"payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.290151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.290112 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.290343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.290170 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rntq\" (UniqueName: \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.291105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.291081 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.314731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.314698 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rntq\" (UniqueName: \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\") pod \"payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.320534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.320511 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.436078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.436043 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea7dd5e1-0470-46d2-8f35-4ddac39b881f" path="/var/lib/kubelet/pods/ea7dd5e1-0470-46d2-8f35-4ddac39b881f/volumes" Oct 30 07:38:10.473826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.473794 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:10.531766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.531738 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:10.632423 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:10.632392 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc705bd40_a916_4f80_a2e7_416cc8f495d2.slice/crio-6ca93d290741a184308785e66c7c08f9bbde68ca6c59d38bec03ad25b0d69e83 WatchSource:0}: Error finding container 6ca93d290741a184308785e66c7c08f9bbde68ca6c59d38bec03ad25b0d69e83: Status 404 returned error can't find the container with id 6ca93d290741a184308785e66c7c08f9bbde68ca6c59d38bec03ad25b0d69e83 Oct 30 07:38:10.633194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.633166 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 07:38:10.779607 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:10.779570 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d33be6_7a14_4c77_b0d8_af05de28a805.slice/crio-4e1b1b22ca008c865e919abf9fe2f7a68df510ecca662874e961509b985b7760 WatchSource:0}: Error finding container 4e1b1b22ca008c865e919abf9fe2f7a68df510ecca662874e961509b985b7760: Status 404 returned error can't find the container with id 4e1b1b22ca008c865e919abf9fe2f7a68df510ecca662874e961509b985b7760 Oct 30 07:38:10.780974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.780951 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 07:38:10.789671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.789614 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" event={"ID":"c705bd40-a916-4f80-a2e7-416cc8f495d2","Type":"ContainerStarted","Data":"6e33d2f746580932fa261ad42dcc5a460c3716ad8b43cbb894587d184d132758"} Oct 30 07:38:10.789788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.789768 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:10.789859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.789802 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" event={"ID":"c705bd40-a916-4f80-a2e7-416cc8f495d2","Type":"ContainerStarted","Data":"6ca93d290741a184308785e66c7c08f9bbde68ca6c59d38bec03ad25b0d69e83"} Oct 30 07:38:10.835108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:10.835051 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" podStartSLOduration=1.835037596 podStartE2EDuration="1.835037596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:10.831935215 +0000 UTC m=+2454.522852654" watchObservedRunningTime="2026-10-30 07:38:10.835037596 +0000 UTC m=+2454.525955046" Oct 30 07:38:11.798545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:11.798508 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" event={"ID":"09d33be6-7a14-4c77-b0d8-af05de28a805","Type":"ContainerStarted","Data":"9e77435235833b5ee1bc022b0773d5b266076f16fab866d6e387b1da436e7491"} Oct 30 07:38:11.799028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:11.798569 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" event={"ID":"09d33be6-7a14-4c77-b0d8-af05de28a805","Type":"ContainerStarted","Data":"4e1b1b22ca008c865e919abf9fe2f7a68df510ecca662874e961509b985b7760"} Oct 30 07:38:11.799028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:11.798791 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:11.841161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:11.841103 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" podStartSLOduration=1.841086934 podStartE2EDuration="1.841086934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:11.839805219 +0000 UTC m=+2455.530722666" watchObservedRunningTime="2026-10-30 07:38:11.841086934 +0000 UTC m=+2455.532004371" Oct 30 07:38:12.314559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:12.314526 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:12.701993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:12.701907 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 07:38:12.701993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:12.701989 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 07:38:14.034160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:14.034127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:15.688970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:15.688914 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:17.333736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:17.333697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:19.057123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:19.057088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:19.089751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:19.081927 5894 scope.go:122] "RemoveContainer" containerID="ed3fce97108153dc46cba4921de67d28f9fdf43a1f6b860b992ba73e097f533a" Oct 30 07:38:20.627867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:20.627824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:20.793595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:20.793567 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:38:21.801735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:21.801701 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 07:38:22.223214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:22.223127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:22.809925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:22.809893 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 07:38:23.943822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:23.943778 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:25.526248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:25.526209 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:27.146819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:27.146785 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:28.860509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:28.860471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:29.796912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:29.796874 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:30.267628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:30.267543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:30.727968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:30.727940 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:31.494476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:31.494443 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:31.669935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:31.669895 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:31.792051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:31.792012 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:32.147268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:32.147157 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:32.268917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:32.268881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:32.455834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:32.455740 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:32.878738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:32.878705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:32.996558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:32.996511 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:33.402374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:33.402331 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:34.175322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.175288 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:38:34.371697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.371666 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:34.498298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.498215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:34.636577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.636546 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:38:34.641132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.641116 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.659857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.659824 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0f76b0\"" Oct 30 07:38:34.663160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.663136 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"routing-overlay\"" Oct 30 07:38:34.663682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.663660 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:38:34.672810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.672785 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"agc-cred-226dc10745b0\"" Oct 30 07:38:34.673197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.673173 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"payload-processing-external-model-e2e--b7f37f47-dockercfg-4w642\"" Oct 30 07:38:34.750615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.750538 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.750789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.750617 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.750789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.750742 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.750901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.750809 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4gjf\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.760116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.760089 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"kube-root-ca.crt\"" Oct 30 07:38:34.760310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.760292 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"openshift-service-ca.crt\"" Oct 30 07:38:34.852853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.851971 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4gjf\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.852853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.852177 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.852853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.852253 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.852853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.852324 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.854780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.854728 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.855812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.855752 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.878973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.878919 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4gjf\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.883383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.883354 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:34.888786 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:38:34.888748 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:38:34.961834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:34.961802 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:35.319750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:35.319716 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:38:35.320121 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:38:35.319842 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ab86b5_d305_436d_bb75_4430f3930c0d.slice/crio-10efefe22be366b3a429c75723a9a3f1ee5a973c7bd1b30a42d20c1c944ae6b5 WatchSource:0}: Error finding container 10efefe22be366b3a429c75723a9a3f1ee5a973c7bd1b30a42d20c1c944ae6b5: Status 404 returned error can't find the container with id 10efefe22be366b3a429c75723a9a3f1ee5a973c7bd1b30a42d20c1c944ae6b5 Oct 30 07:38:35.986260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:35.986222 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" event={"ID":"f1ab86b5-d305-436d-bb75-4430f3930c0d","Type":"ContainerStarted","Data":"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b"} Oct 30 07:38:35.986451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:35.986285 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" event={"ID":"f1ab86b5-d305-436d-bb75-4430f3930c0d","Type":"ContainerStarted","Data":"10efefe22be366b3a429c75723a9a3f1ee5a973c7bd1b30a42d20c1c944ae6b5"} Oct 30 07:38:35.986451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:35.986306 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:36.029857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:36.029795 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" podStartSLOduration=2.029780965 podStartE2EDuration="2.029780965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:38:36.028950122 +0000 UTC m=+2479.719867559" watchObservedRunningTime="2026-10-30 07:38:36.029780965 +0000 UTC m=+2479.720698402" Oct 30 07:38:36.162087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:36.161914 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:36.325273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:36.325236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:37.892031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:37.892000 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:38.015796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:38.015761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:38.139012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:38.138969 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:39.619627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:39.619594 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:41.327597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:41.327561 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:43.042579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:43.042542 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:44.805350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:44.805308 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:46.516327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:46.516288 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:46.998270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:46.998241 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:38:48.108759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:48.108725 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:48.228834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:48.228799 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:48.340017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:48.339979 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:49.880862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:49.880813 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:51.636672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:51.636614 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:53.350570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:53.350533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:55.032443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:55.032396 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:56.791398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:56.791360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:38:58.519721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:38:58.519680 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:00.219971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:00.219932 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:01.939359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:01.939314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:03.526537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:03.526505 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:04.223394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.223356 5894 generic.go:356] "Generic (PLEG): container finished" podID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerID="5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9" exitCode=0 Oct 30 07:39:04.223666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.223431 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerDied","Data":"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9"} Oct 30 07:39:04.223951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.223934 5894 scope.go:122] "RemoveContainer" containerID="5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9" Oct 30 07:39:04.226049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.226017 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:04.226124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.226107 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:04.226165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:04.226152 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:39:04.832885 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:39:04.832842 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:39:05.210176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:05.210079 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:05.232136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:05.232107 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerStarted","Data":"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed"} Oct 30 07:39:05.901076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:05.901031 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:39:05.901627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:05.901451 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:39:05.901824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:05.901544 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 07:39:06.901209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:06.901174 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:39:06.901731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:06.901246 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 07:39:07.018055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:07.018025 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:07.901256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:07.901217 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:39:07.901754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:07.901278 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 07:39:08.751571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:08.751530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:08.901659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:08.901247 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:39:08.901659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:08.901327 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 07:39:09.901059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:09.901023 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body= Oct 30 07:39:09.901382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:09.901354 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" Oct 30 07:39:10.556098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:10.556067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:10.907003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:10.906898 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:39:10.907361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:10.907318 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:39:10.907992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:10.907973 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 07:39:12.153532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:12.153498 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:13.799247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:13.799216 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:15.452382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:15.452341 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:17.146333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:17.146299 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:18.731489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:18.731449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:20.325178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:20.325143 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:22.023239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:22.023202 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:22.249780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:22.249744 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:39:23.905665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:23.905621 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:25.016304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.016249 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:25.594514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.594479 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:25.748125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.748081 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g"] Oct 30 07:39:25.752861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.752836 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.773766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.773736 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g"] Oct 30 07:39:25.781562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.781536 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:39:25.781762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.781701 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:39:25.781762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.781733 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:39:25.781875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.781802 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:39:25.807692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.807662 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.807879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.807713 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9csth\" (UniqueName: \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.807879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.807744 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.908766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.908666 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.908766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.908724 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9csth\" (UniqueName: \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.908766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.908750 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.909541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.909517 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.911360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.911331 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:25.949318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:25.949286 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9csth\" (UniqueName: \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\") pod \"maas-api-revoke-sub-62fc22bb-wqm2g\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:26.067840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.067805 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:26.162265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.162182 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v"] Oct 30 07:39:26.167803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.167779 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.189903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.189870 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v"] Oct 30 07:39:26.212749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.212720 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.212922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.212755 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27k2l\" (UniqueName: \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.213051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.213028 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.314753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.314719 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.314957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.314794 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.314957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.314827 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k2l\" (UniqueName: \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.315542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.315512 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.317392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.317366 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.338442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.338416 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27k2l\" (UniqueName: \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\") pod \"maas-api-revoke-sub-ab41498c-4mv5v\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.378524 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:26.378488 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c057c43_7296_4802_9b64_d346fe396036.slice/crio-b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d WatchSource:0}: Error finding container b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d: Status 404 returned error can't find the container with id b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d Oct 30 07:39:26.379429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.379409 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g"] Oct 30 07:39:26.387695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.387662 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" event={"ID":"8c057c43-7296-4802-9b64-d346fe396036","Type":"ContainerStarted","Data":"b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d"} Oct 30 07:39:26.403536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.403514 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:26.489358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.489321 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:26.873092 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:26.872736 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42957b4b_773d_4ebc_a05b_774678b27ab7.slice/crio-d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f WatchSource:0}: Error finding container d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f: Status 404 returned error can't find the container with id d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f Oct 30 07:39:26.874532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.874495 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v"] Oct 30 07:39:26.886118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:26.886086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:27.270532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.270496 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:27.396714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.396673 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" event={"ID":"8c057c43-7296-4802-9b64-d346fe396036","Type":"ContainerStarted","Data":"d265f31e66c25a0d8eab76c3daf29f668a73d908ab651063b20cb5858f6aeeb8"} Oct 30 07:39:27.398857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.398830 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" event={"ID":"42957b4b-773d-4ebc-a05b-774678b27ab7","Type":"ContainerStarted","Data":"dc89b07b1508192431e0b866aeb7d0319cf850f4a19af057ba8dc4d5f7d4959d"} Oct 30 07:39:27.398986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.398885 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" event={"ID":"42957b4b-773d-4ebc-a05b-774678b27ab7","Type":"ContainerStarted","Data":"d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f"} Oct 30 07:39:27.456361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.456258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:27.463287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.463227 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" podStartSLOduration=2.463208954 podStartE2EDuration="2.463208954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:27.462890132 +0000 UTC m=+2531.153807571" watchObservedRunningTime="2026-10-30 07:39:27.463208954 +0000 UTC m=+2531.154126391" Oct 30 07:39:27.511293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:27.511243 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" podStartSLOduration=1.511230674 podStartE2EDuration="1.511230674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:27.510903038 +0000 UTC m=+2531.201820475" watchObservedRunningTime="2026-10-30 07:39:27.511230674 +0000 UTC m=+2531.202148110" Oct 30 07:39:28.046307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:28.046213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:28.407750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:28.407624 5894 generic.go:356] "Generic (PLEG): container finished" podID="42957b4b-773d-4ebc-a05b-774678b27ab7" containerID="dc89b07b1508192431e0b866aeb7d0319cf850f4a19af057ba8dc4d5f7d4959d" exitCode=0 Oct 30 07:39:28.407750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:28.407687 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" event={"ID":"42957b4b-773d-4ebc-a05b-774678b27ab7","Type":"ContainerDied","Data":"dc89b07b1508192431e0b866aeb7d0319cf850f4a19af057ba8dc4d5f7d4959d"} Oct 30 07:39:28.409666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:28.409628 5894 generic.go:356] "Generic (PLEG): container finished" podID="8c057c43-7296-4802-9b64-d346fe396036" containerID="d265f31e66c25a0d8eab76c3daf29f668a73d908ab651063b20cb5858f6aeeb8" exitCode=0 Oct 30 07:39:28.409795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:28.409663 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" event={"ID":"8c057c43-7296-4802-9b64-d346fe396036","Type":"ContainerDied","Data":"d265f31e66c25a0d8eab76c3daf29f668a73d908ab651063b20cb5858f6aeeb8"} Oct 30 07:39:29.072348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.072310 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:29.587966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.587937 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:29.633220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.633191 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:29.665971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.665935 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\" (UniqueName: \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\") pod \"42957b4b-773d-4ebc-a05b-774678b27ab7\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " Oct 30 07:39:29.666165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666043 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\") pod \"8c057c43-7296-4802-9b64-d346fe396036\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " Oct 30 07:39:29.666165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666115 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\") pod \"8c057c43-7296-4802-9b64-d346fe396036\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " Oct 30 07:39:29.666296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666239 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\" (UniqueName: \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\") pod \"8c057c43-7296-4802-9b64-d346fe396036\" (UID: \"8c057c43-7296-4802-9b64-d346fe396036\") " Oct 30 07:39:29.666358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666331 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\") pod \"42957b4b-773d-4ebc-a05b-774678b27ab7\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " Oct 30 07:39:29.667112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666455 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\") pod \"42957b4b-773d-4ebc-a05b-774678b27ab7\" (UID: \"42957b4b-773d-4ebc-a05b-774678b27ab7\") " Oct 30 07:39:29.667112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.666478 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca" pod "8c057c43-7296-4802-9b64-d346fe396036" (UID: "8c057c43-7296-4802-9b64-d346fe396036"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:29.667112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.667036 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c057c43-7296-4802-9b64-d346fe396036-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.667112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.667063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca" pod "42957b4b-773d-4ebc-a05b-774678b27ab7" (UID: "42957b4b-773d-4ebc-a05b-774678b27ab7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:29.668602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.668563 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l" pod "42957b4b-773d-4ebc-a05b-774678b27ab7" (UID: "42957b4b-773d-4ebc-a05b-774678b27ab7"). InnerVolumeSpecName "kube-api-access-27k2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:29.669129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.669095 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca" pod "8c057c43-7296-4802-9b64-d346fe396036" (UID: "8c057c43-7296-4802-9b64-d346fe396036"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:39:29.669227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.669153 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca" pod "42957b4b-773d-4ebc-a05b-774678b27ab7" (UID: "42957b4b-773d-4ebc-a05b-774678b27ab7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:39:29.669291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.669219 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth" pod "8c057c43-7296-4802-9b64-d346fe396036" (UID: "8c057c43-7296-4802-9b64-d346fe396036"). InnerVolumeSpecName "kube-api-access-9csth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:29.768779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.768737 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.768779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.768772 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42957b4b-773d-4ebc-a05b-774678b27ab7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.768779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.768783 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27k2l\" (UniqueName: \"kubernetes.io/projected/42957b4b-773d-4ebc-a05b-774678b27ab7-kube-api-access-27k2l\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.769070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.768793 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c057c43-7296-4802-9b64-d346fe396036-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.769070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.768802 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9csth\" (UniqueName: \"kubernetes.io/projected/8c057c43-7296-4802-9b64-d346fe396036-kube-api-access-9csth\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:29.914527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:29.914493 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:30.038987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.038939 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt"] Oct 30 07:39:30.045251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.045218 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4ded9bc-2k6bt"] Oct 30 07:39:30.425125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.425040 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" Oct 30 07:39:30.426624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.426603 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" Oct 30 07:39:30.434890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.434864 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5e2cc6-11c4-4960-ba19-90249a64d88e" path="/var/lib/kubelet/pods/ac5e2cc6-11c4-4960-ba19-90249a64d88e/volumes" Oct 30 07:39:30.435701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.435672 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g" event={"ID":"8c057c43-7296-4802-9b64-d346fe396036","Type":"ContainerDied","Data":"b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d"} Oct 30 07:39:30.435935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.435727 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7210a7ad9c0b4ccb6257bfba2f38436a254710f303c33ba0499e17a04239e9d" Oct 30 07:39:30.435935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.435743 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v" event={"ID":"42957b4b-773d-4ebc-a05b-774678b27ab7","Type":"ContainerDied","Data":"d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f"} Oct 30 07:39:30.435935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.435768 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3135556050389cc236c6bb4d703da399a3741951eed2833200cf58a6953135f" Oct 30 07:39:30.764221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:30.764132 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:31.033911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:31.033882 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc"] Oct 30 07:39:31.036878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:31.036856 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afe72f5d-hrgfc"] Oct 30 07:39:31.428834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:31.428728 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:31.551388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:31.551350 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:32.432946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:32.432916 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8692621e-332b-47dc-9b56-ba297c3a7c95" path="/var/lib/kubelet/pods/8692621e-332b-47dc-9b56-ba297c3a7c95/volumes" Oct 30 07:39:32.535137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:32.535104 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:34.360320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.360289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:34.490119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.490079 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:34.510431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.510394 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 07:39:34.511150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511134 5894 cpu_manager.go:414] "containerMap: removing container" podUID="42957b4b-773d-4ebc-a05b-774678b27ab7" containerName="revoke-keys" Oct 30 07:39:34.511198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511151 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42957b4b-773d-4ebc-a05b-774678b27ab7" containerName="revoke-keys" Oct 30 07:39:34.511198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511164 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8c057c43-7296-4802-9b64-d346fe396036" containerName="revoke-keys" Oct 30 07:39:34.511198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511170 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c057c43-7296-4802-9b64-d346fe396036" containerName="revoke-keys" Oct 30 07:39:34.511331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511297 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="42957b4b-773d-4ebc-a05b-774678b27ab7" containerName="revoke-keys" Oct 30 07:39:34.511331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.511307 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c057c43-7296-4802-9b64-d346fe396036" containerName="revoke-keys" Oct 30 07:39:34.516475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.516448 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.541193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.541164 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 07:39:34.549883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.549855 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bb2e8f\"" Oct 30 07:39:34.549883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.549854 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bb2e8f-dockercfg-lfclh\"" Oct 30 07:39:34.560790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.560755 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 07:39:34.565238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.565217 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.587068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.587034 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 07:39:34.620923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.620832 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:34.641972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.640331 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrdz\" (UniqueName: \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.641972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.640483 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cqr\" (UniqueName: \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.641972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.640599 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.641972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.640701 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.741857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.741808 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cqr\" (UniqueName: \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.742018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.741937 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.742081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.742017 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.742125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.742116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrdz\" (UniqueName: \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.743026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.742980 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.743026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.743014 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.768127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.768093 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrdz\" (UniqueName: \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\") pod \"payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.768297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.768093 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6cqr\" (UniqueName: \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\") pod \"payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:34.830929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.830887 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:34.831169 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:39:34.831131 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:39:34.895025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:34.894532 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:35.183175 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:35.183147 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a864dd_601f_426a_beca_2a039bb0e46b.slice/crio-66c45437318ff87e15f17d7142c5e019fe26db69d066416bb0dbc534f0b14a25 WatchSource:0}: Error finding container 66c45437318ff87e15f17d7142c5e019fe26db69d066416bb0dbc534f0b14a25: Status 404 returned error can't find the container with id 66c45437318ff87e15f17d7142c5e019fe26db69d066416bb0dbc534f0b14a25 Oct 30 07:39:35.184073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.184051 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 07:39:35.241761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.241734 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 07:39:35.248204 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:35.248179 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6895f08d_18b4_40ed_a137_891eb9f6dfb6.slice/crio-908c263e5d0dda0634a03eed953b5c4114a5ca1162e41b6ba7e72599cae0a4c9 WatchSource:0}: Error finding container 908c263e5d0dda0634a03eed953b5c4114a5ca1162e41b6ba7e72599cae0a4c9: Status 404 returned error can't find the container with id 908c263e5d0dda0634a03eed953b5c4114a5ca1162e41b6ba7e72599cae0a4c9 Oct 30 07:39:35.476028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.475935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" event={"ID":"6895f08d-18b4-40ed-a137-891eb9f6dfb6","Type":"ContainerStarted","Data":"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36"} Oct 30 07:39:35.476028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.476020 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" event={"ID":"6895f08d-18b4-40ed-a137-891eb9f6dfb6","Type":"ContainerStarted","Data":"908c263e5d0dda0634a03eed953b5c4114a5ca1162e41b6ba7e72599cae0a4c9"} Oct 30 07:39:35.476527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.476058 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:35.479582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.479545 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" event={"ID":"59a864dd-601f-426a-beca-2a039bb0e46b","Type":"ContainerStarted","Data":"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca"} Oct 30 07:39:35.479751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.479596 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" event={"ID":"59a864dd-601f-426a-beca-2a039bb0e46b","Type":"ContainerStarted","Data":"66c45437318ff87e15f17d7142c5e019fe26db69d066416bb0dbc534f0b14a25"} Oct 30 07:39:35.479751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.479625 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:35.523977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.523927 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" podStartSLOduration=1.523911473 podStartE2EDuration="1.523911473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:35.522098168 +0000 UTC m=+2539.213015605" watchObservedRunningTime="2026-10-30 07:39:35.523911473 +0000 UTC m=+2539.214828910" Oct 30 07:39:35.571388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.571334 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" podStartSLOduration=1.571319833 podStartE2EDuration="1.571319833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:35.568614851 +0000 UTC m=+2539.259532288" watchObservedRunningTime="2026-10-30 07:39:35.571319833 +0000 UTC m=+2539.262237270" Oct 30 07:39:35.823189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.823145 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:35.949769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:35.949727 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:36.114103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:36.114015 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:36.239958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:36.239924 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:36.575865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:36.575828 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:36.709887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:36.709841 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:37.280685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:37.280610 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:37.393587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:37.393549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:38.156946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.156851 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:38.278241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.278199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:38.405075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.405033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:38.421989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.421911 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:39:38.524607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.524549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:38.850425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:38.850385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:39.040059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:39.040019 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:39.398773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:39.398742 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:39:39.478939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:39.478901 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:39.946582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:39.946551 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:40.754460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:40.754423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:41.003889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.003854 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:39:41.008947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.008890 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.038468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.038430 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-e2e--40e0c8c8-dockercfg-mf25w\"" Oct 30 07:39:41.038712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.038475 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b3a7d7\"" Oct 30 07:39:41.038712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.038531 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"agc-cred-e6a29ec0d8f8\"" Oct 30 07:39:41.038712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.038585 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"routing-overlay\"" Oct 30 07:39:41.038712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.038679 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"openshift-service-ca.crt\"" Oct 30 07:39:41.039087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.039059 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"kube-root-ca.crt\"" Oct 30 07:39:41.042559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.042515 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:39:41.045924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.045899 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5"] Oct 30 07:39:41.054931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.054905 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b9e2d77f-lh4h5"] Oct 30 07:39:41.144368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.144329 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.144575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.144468 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.144575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.144524 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.144710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.144693 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpz9p\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.246560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.246515 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpz9p\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.246882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.246830 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.246995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.246940 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.247127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.247003 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.247927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.247863 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.247927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.247866 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.251280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.251244 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.289451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.289414 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpz9p\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.327667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.325878 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:41.644686 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:41.644615 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161ab4fa_123c_4b7e_826e_9c8587bb003c.slice/crio-05dd83d8ea4d948f1968eba7687070832d1b2f3682ccc12b40353cba5a3929c8 WatchSource:0}: Error finding container 05dd83d8ea4d948f1968eba7687070832d1b2f3682ccc12b40353cba5a3929c8: Status 404 returned error can't find the container with id 05dd83d8ea4d948f1968eba7687070832d1b2f3682ccc12b40353cba5a3929c8 Oct 30 07:39:41.644903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.644884 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:39:41.692406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:41.692379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:42.043984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.043948 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9"] Oct 30 07:39:42.055804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.055768 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8660fdf4-txvm9"] Oct 30 07:39:42.433532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.433451 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660dc935-2e30-4bb6-8dd1-871f11baef20" path="/var/lib/kubelet/pods/660dc935-2e30-4bb6-8dd1-871f11baef20/volumes" Oct 30 07:39:42.433968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.433952 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99718b6-8ba5-498a-8eca-2bd952168174" path="/var/lib/kubelet/pods/b99718b6-8ba5-498a-8eca-2bd952168174/volumes" Oct 30 07:39:42.551736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.551692 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" event={"ID":"161ab4fa-123c-4b7e-826e-9c8587bb003c","Type":"ContainerStarted","Data":"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b"} Oct 30 07:39:42.551944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.551755 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" event={"ID":"161ab4fa-123c-4b7e-826e-9c8587bb003c","Type":"ContainerStarted","Data":"05dd83d8ea4d948f1968eba7687070832d1b2f3682ccc12b40353cba5a3929c8"} Oct 30 07:39:42.551944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.551903 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:42.594921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:42.594862 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" podStartSLOduration=2.594848328 podStartE2EDuration="2.594848328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:42.59375702 +0000 UTC m=+2546.284674458" watchObservedRunningTime="2026-10-30 07:39:42.594848328 +0000 UTC m=+2546.285765765" Oct 30 07:39:43.330082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.330044 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:43.422694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.422629 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" podUID="a33622ca-6b90-46c5-904e-c4112bb4aabe" containerName="payload-processing" containerID="cri-o://ff5b8cc1f36b0579c2c23825d307141804d7960b7487e2a0439017a0cabedf6d" gracePeriod=25 Oct 30 07:39:43.560939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.560905 5894 generic.go:356] "Generic (PLEG): container finished" podID="a33622ca-6b90-46c5-904e-c4112bb4aabe" containerID="ff5b8cc1f36b0579c2c23825d307141804d7960b7487e2a0439017a0cabedf6d" exitCode=0 Oct 30 07:39:43.561112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.560953 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" event={"ID":"a33622ca-6b90-46c5-904e-c4112bb4aabe","Type":"ContainerDied","Data":"ff5b8cc1f36b0579c2c23825d307141804d7960b7487e2a0439017a0cabedf6d"} Oct 30 07:39:43.699424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.699395 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:39:43.883975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.883886 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\") pod \"a33622ca-6b90-46c5-904e-c4112bb4aabe\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " Oct 30 07:39:43.883975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.883934 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\") pod \"a33622ca-6b90-46c5-904e-c4112bb4aabe\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " Oct 30 07:39:43.884226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884020 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\") pod \"a33622ca-6b90-46c5-904e-c4112bb4aabe\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " Oct 30 07:39:43.884226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884200 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\") pod \"a33622ca-6b90-46c5-904e-c4112bb4aabe\" (UID: \"a33622ca-6b90-46c5-904e-c4112bb4aabe\") " Oct 30 07:39:43.884402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884351 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume" pod "a33622ca-6b90-46c5-904e-c4112bb4aabe" (UID: "a33622ca-6b90-46c5-904e-c4112bb4aabe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:43.884625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884600 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume" pod "a33622ca-6b90-46c5-904e-c4112bb4aabe" (UID: "a33622ca-6b90-46c5-904e-c4112bb4aabe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:43.884875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884852 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:43.884875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.884875 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a33622ca-6b90-46c5-904e-c4112bb4aabe-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:43.886381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.886359 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials" pod "a33622ca-6b90-46c5-904e-c4112bb4aabe" (UID: "a33622ca-6b90-46c5-904e-c4112bb4aabe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:43.886484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.886402 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz" pod "a33622ca-6b90-46c5-904e-c4112bb4aabe" (UID: "a33622ca-6b90-46c5-904e-c4112bb4aabe"). InnerVolumeSpecName "kube-api-access-qgfsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:43.986236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.986201 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:43.986236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:43.986230 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgfsz\" (UniqueName: \"kubernetes.io/projected/a33622ca-6b90-46c5-904e-c4112bb4aabe-kube-api-access-qgfsz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:44.399073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.399024 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerName="payload-processing" containerID="cri-o://629ef0d36aa8588b99616e98c75daa93f1370e5ad3327150d5c4cecf5190eeb9" gracePeriod=25 Oct 30 07:39:44.538251 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:44.538218 5894 logging.go:55] [core] [Channel #1508 SubChannel #1509]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.226:50052", ServerName: "10.134.0.226:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.226:50052: connect: connection refused" Oct 30 07:39:44.570434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.570406 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" Oct 30 07:39:44.570650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.570435 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d" event={"ID":"a33622ca-6b90-46c5-904e-c4112bb4aabe","Type":"ContainerDied","Data":"8f11ef6af9e3d6a4a7b94f3fb6b4436a4f5f4801ce5c3ca28fc0488236be771b"} Oct 30 07:39:44.570650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.570504 5894 scope.go:122] "RemoveContainer" containerID="ff5b8cc1f36b0579c2c23825d307141804d7960b7487e2a0439017a0cabedf6d" Oct 30 07:39:44.573084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.573056 5894 generic.go:356] "Generic (PLEG): container finished" podID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerID="629ef0d36aa8588b99616e98c75daa93f1370e5ad3327150d5c4cecf5190eeb9" exitCode=0 Oct 30 07:39:44.573205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.573145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" event={"ID":"74935ffd-8a09-4d3b-99a5-f5f875748e9c","Type":"ContainerDied","Data":"629ef0d36aa8588b99616e98c75daa93f1370e5ad3327150d5c4cecf5190eeb9"} Oct 30 07:39:44.626880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.626848 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:39:44.641623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.641552 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-7454cqn2d"] Oct 30 07:39:44.704145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.704115 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:39:44.800922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.800885 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\") pod \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " Oct 30 07:39:44.801127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801006 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\") pod \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " Oct 30 07:39:44.801127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801050 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\") pod \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " Oct 30 07:39:44.801257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801150 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\") pod \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\" (UID: \"74935ffd-8a09-4d3b-99a5-f5f875748e9c\") " Oct 30 07:39:44.801441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801414 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume" pod "74935ffd-8a09-4d3b-99a5-f5f875748e9c" (UID: "74935ffd-8a09-4d3b-99a5-f5f875748e9c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:44.801529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801499 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume" pod "74935ffd-8a09-4d3b-99a5-f5f875748e9c" (UID: "74935ffd-8a09-4d3b-99a5-f5f875748e9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:39:44.801736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801712 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:44.801798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.801740 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74935ffd-8a09-4d3b-99a5-f5f875748e9c-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:44.803303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.803282 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials" pod "74935ffd-8a09-4d3b-99a5-f5f875748e9c" (UID: "74935ffd-8a09-4d3b-99a5-f5f875748e9c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:44.803374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.803338 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh" pod "74935ffd-8a09-4d3b-99a5-f5f875748e9c" (UID: "74935ffd-8a09-4d3b-99a5-f5f875748e9c"). InnerVolumeSpecName "kube-api-access-dc4bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:39:44.903532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.903493 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:44.903532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:44.903525 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4bh\" (UniqueName: \"kubernetes.io/projected/74935ffd-8a09-4d3b-99a5-f5f875748e9c-kube-api-access-dc4bh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:39:45.006817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.006725 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:45.538337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.538286 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.226:50052\" within 1s: context deadline exceeded" Oct 30 07:39:45.584289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.584259 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" Oct 30 07:39:45.584487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.584258 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h" event={"ID":"74935ffd-8a09-4d3b-99a5-f5f875748e9c","Type":"ContainerDied","Data":"5c948f73bb2528a2af591899658ed2d07b5edcadd8083984e71e1cb2fe4713d6"} Oct 30 07:39:45.584487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.584400 5894 scope.go:122] "RemoveContainer" containerID="629ef0d36aa8588b99616e98c75daa93f1370e5ad3327150d5c4cecf5190eeb9" Oct 30 07:39:45.604104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.604069 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:45.639781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.639738 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:39:45.651802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.651758 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e193ee/payload-processing-external-model-e2e-extmod-b-e193ee-5bccfdz2h"] Oct 30 07:39:45.729563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.729526 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:45.845714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.843474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:45.959322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:45.959283 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:46.432694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:46.432629 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" path="/var/lib/kubelet/pods/74935ffd-8a09-4d3b-99a5-f5f875748e9c/volumes" Oct 30 07:39:46.433114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:46.433100 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33622ca-6b90-46c5-904e-c4112bb4aabe" path="/var/lib/kubelet/pods/a33622ca-6b90-46c5-904e-c4112bb4aabe/volumes" Oct 30 07:39:46.493083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:46.493049 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 07:39:46.493308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:46.493197 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 07:39:46.598204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:46.598154 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:48.406614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:48.406573 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:49.890740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:49.890482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:50.009239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:50.009193 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:50.459400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:50.459364 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:51.313676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.313616 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:39:51.314548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314525 5894 cpu_manager.go:414] "containerMap: removing container" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerName="payload-processing" Oct 30 07:39:51.314685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314549 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerName="payload-processing" Oct 30 07:39:51.314685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314592 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a33622ca-6b90-46c5-904e-c4112bb4aabe" containerName="payload-processing" Oct 30 07:39:51.314685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314600 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33622ca-6b90-46c5-904e-c4112bb4aabe" containerName="payload-processing" Oct 30 07:39:51.314860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314803 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33622ca-6b90-46c5-904e-c4112bb4aabe" containerName="payload-processing" Oct 30 07:39:51.314860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.314835 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="74935ffd-8a09-4d3b-99a5-f5f875748e9c" containerName="payload-processing" Oct 30 07:39:51.319854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.319830 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.336675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.336648 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"agc-cred-6459ba1f96f0\"" Oct 30 07:39:51.341561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.341527 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:39:51.411724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.411683 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.411926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.411773 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7st\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.411926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.411846 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.412010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.411987 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.513116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.513065 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.513302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.513146 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7st\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.513302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.513292 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.513796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.513764 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.513990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.513959 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.514329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.514308 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.516346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.516322 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.536771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.536733 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7st\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.634355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.634267 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:51.942808 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:51.942771 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f32bf0_fd7c_4ca3_a770_04236a20651f.slice/crio-a9ead08c46102d6fb4baf25f6e17ae3e17c319725aaafe71f26728ad4c5eb292 WatchSource:0}: Error finding container a9ead08c46102d6fb4baf25f6e17ae3e17c319725aaafe71f26728ad4c5eb292: Status 404 returned error can't find the container with id a9ead08c46102d6fb4baf25f6e17ae3e17c319725aaafe71f26728ad4c5eb292 Oct 30 07:39:51.943627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:51.943602 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:39:52.277942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.277911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:52.578997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.578911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:52.650417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.650379 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" event={"ID":"31f32bf0-fd7c-4ca3-a770-04236a20651f","Type":"ContainerStarted","Data":"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf"} Oct 30 07:39:52.650616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.650442 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" event={"ID":"31f32bf0-fd7c-4ca3-a770-04236a20651f","Type":"ContainerStarted","Data":"a9ead08c46102d6fb4baf25f6e17ae3e17c319725aaafe71f26728ad4c5eb292"} Oct 30 07:39:52.650616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.650466 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:39:52.709808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.709757 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" podStartSLOduration=1.709741698 podStartE2EDuration="1.709741698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:52.706759081 +0000 UTC m=+2556.397676519" watchObservedRunningTime="2026-10-30 07:39:52.709741698 +0000 UTC m=+2556.400659134" Oct 30 07:39:52.780538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.780494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:52.978977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:52.978894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:53.103247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:53.103199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:53.247740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:53.244447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:53.358400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:53.358364 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:53.543252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:53.543178 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:53.584928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:53.584897 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:39:54.324905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:54.324844 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:54.460697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:54.460658 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:54.982223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:54.982186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:55.259271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.259197 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:39:55.264118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.264093 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.283284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.283258 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"kube-root-ca.crt\"" Oct 30 07:39:55.286355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.286326 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:39:55.294491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.294464 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"openshift-service-ca.crt\"" Oct 30 07:39:55.294491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.294468 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"routing-overlay\"" Oct 30 07:39:55.294767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.294464 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-plugins-e2e-extmod-a-705337\"" Oct 30 07:39:55.367231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.367199 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"e2e-cmp-legacy-bf3b3e-provider-cred\"" Oct 30 07:39:55.367421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.367245 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-e2e--90fb40dc-dockercfg-sgjh6\"" Oct 30 07:39:55.381940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.381901 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.382756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.381992 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg9p9\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.382756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.382168 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.382756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.382242 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.484094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.484052 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.484356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.484117 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg9p9\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.484436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.484361 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.484497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.484443 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.485237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.485207 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.485396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.485295 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.487517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.487490 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.508676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.508612 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg9p9\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\") pod \"payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.579560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.579516 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:55.897624 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:55.897586 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0236897a_10f8_43fe_bb88_3cd1f26de668.slice/crio-aec2a48e7bae75a9cbdbacf880515ab863b3b0ecea1da8a7419fd4e7873a9db1 WatchSource:0}: Error finding container aec2a48e7bae75a9cbdbacf880515ab863b3b0ecea1da8a7419fd4e7873a9db1: Status 404 returned error can't find the container with id aec2a48e7bae75a9cbdbacf880515ab863b3b0ecea1da8a7419fd4e7873a9db1 Oct 30 07:39:55.898006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:55.897920 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:39:56.033092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.033056 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:56.204299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.204207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:56.286986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.286948 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:39:56.291474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.291453 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.312171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.312137 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"openshift-service-ca.crt\"" Oct 30 07:39:56.312171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.312183 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"e2e-cmp-legacy-5b673d-provider-cred\"" Oct 30 07:39:56.321775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.321741 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"kube-root-ca.crt\"" Oct 30 07:39:56.331248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.331194 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1e0880\"" Oct 30 07:39:56.331957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.331357 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"routing-overlay\"" Oct 30 07:39:56.332163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.332140 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"payload-processing-external-model-e2e--cc0b775b-dockercfg-5rvhc\"" Oct 30 07:39:56.341655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.339282 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:39:56.402234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.402189 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzjz\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.402427 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.402292 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.402508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.402442 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.402590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.402565 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.506176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.506081 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.508101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.506861 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.508101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.506961 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.508101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.507211 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzjz\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.508101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.507583 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.508445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.508383 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.511816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.511263 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.532241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.532206 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzjz\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.608086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.608045 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:56.694054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.694012 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" event={"ID":"0236897a-10f8-43fe-bb88-3cd1f26de668","Type":"ContainerStarted","Data":"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d"} Oct 30 07:39:56.694245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.694082 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" event={"ID":"0236897a-10f8-43fe-bb88-3cd1f26de668","Type":"ContainerStarted","Data":"aec2a48e7bae75a9cbdbacf880515ab863b3b0ecea1da8a7419fd4e7873a9db1"} Oct 30 07:39:56.694245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.694127 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:39:56.737610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.737559 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" podStartSLOduration=1.737546455 podStartE2EDuration="1.737546455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:56.736719195 +0000 UTC m=+2560.427636634" watchObservedRunningTime="2026-10-30 07:39:56.737546455 +0000 UTC m=+2560.428463893" Oct 30 07:39:56.919141 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:39:56.919103 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b75f30_f708_41d9_8ef9_0f57db93ef67.slice/crio-62f4f7cf901b084a6e50bb6af3d86ddf4508f4afdd84315f5e54c1fce7826f7b WatchSource:0}: Error finding container 62f4f7cf901b084a6e50bb6af3d86ddf4508f4afdd84315f5e54c1fce7826f7b: Status 404 returned error can't find the container with id 62f4f7cf901b084a6e50bb6af3d86ddf4508f4afdd84315f5e54c1fce7826f7b Oct 30 07:39:56.920198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:56.920153 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:39:57.703156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:57.703114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" event={"ID":"24b75f30-f708-41d9-8ef9-0f57db93ef67","Type":"ContainerStarted","Data":"ec20c0af3d9227266313f18fb33989aa8d5243884a8f09c6db58edb53867d851"} Oct 30 07:39:57.703698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:57.703180 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" event={"ID":"24b75f30-f708-41d9-8ef9-0f57db93ef67","Type":"ContainerStarted","Data":"62f4f7cf901b084a6e50bb6af3d86ddf4508f4afdd84315f5e54c1fce7826f7b"} Oct 30 07:39:57.703698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:57.703208 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:39:57.749798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:57.749742 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" podStartSLOduration=1.7497280929999999 podStartE2EDuration="1.749728093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:39:57.748318049 +0000 UTC m=+2561.439235487" watchObservedRunningTime="2026-10-30 07:39:57.749728093 +0000 UTC m=+2561.440645529" Oct 30 07:39:57.799414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:57.799375 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:39:59.533839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:39:59.533808 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:00.032455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:00.032423 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj"] Oct 30 07:40:00.043946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:00.043917 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3ac01c3-hhswj"] Oct 30 07:40:00.432629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:00.432546 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e28d84-f243-479f-9403-6fb9e0818912" path="/var/lib/kubelet/pods/d4e28d84-f243-479f-9403-6fb9e0818912/volumes" Oct 30 07:40:01.199191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:01.199161 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:02.872691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:02.872626 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:03.667101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:03.667076 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:40:03.742273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:03.742241 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:40:04.528098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:04.528062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:04.830913 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:40:04.830815 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:40:06.112741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:06.112697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:07.706855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:07.706827 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:40:07.775514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:07.775479 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:08.714112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:08.714082 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:40:08.743321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:08.743286 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" podUID="f1ab86b5-d305-436d-bb75-4430f3930c0d" containerName="payload-processing" containerID="cri-o://56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b" gracePeriod=25 Oct 30 07:40:09.015133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.015105 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:40:09.176111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176074 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\") pod \"f1ab86b5-d305-436d-bb75-4430f3930c0d\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " Oct 30 07:40:09.176306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176125 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\") pod \"f1ab86b5-d305-436d-bb75-4430f3930c0d\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " Oct 30 07:40:09.176306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176208 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\") pod \"f1ab86b5-d305-436d-bb75-4430f3930c0d\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " Oct 30 07:40:09.176439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176326 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\") pod \"f1ab86b5-d305-436d-bb75-4430f3930c0d\" (UID: \"f1ab86b5-d305-436d-bb75-4430f3930c0d\") " Oct 30 07:40:09.176691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176657 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume" pod "f1ab86b5-d305-436d-bb75-4430f3930c0d" (UID: "f1ab86b5-d305-436d-bb75-4430f3930c0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:09.176828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176670 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume" pod "f1ab86b5-d305-436d-bb75-4430f3930c0d" (UID: "f1ab86b5-d305-436d-bb75-4430f3930c0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:09.176885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176874 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:09.176925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.176888 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ab86b5-d305-436d-bb75-4430f3930c0d-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:09.179149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.179127 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials" pod "f1ab86b5-d305-436d-bb75-4430f3930c0d" (UID: "f1ab86b5-d305-436d-bb75-4430f3930c0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:09.179248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.179163 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf" pod "f1ab86b5-d305-436d-bb75-4430f3930c0d" (UID: "f1ab86b5-d305-436d-bb75-4430f3930c0d"). InnerVolumeSpecName "kube-api-access-r4gjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:09.278367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.278334 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4gjf\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-kube-api-access-r4gjf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:09.278367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.278365 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ab86b5-d305-436d-bb75-4430f3930c0d-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:09.444532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.444497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:09.814223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.814188 5894 generic.go:356] "Generic (PLEG): container finished" podID="f1ab86b5-d305-436d-bb75-4430f3930c0d" containerID="56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b" exitCode=0 Oct 30 07:40:09.814747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.814237 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" Oct 30 07:40:09.814747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.814272 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" event={"ID":"f1ab86b5-d305-436d-bb75-4430f3930c0d","Type":"ContainerDied","Data":"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b"} Oct 30 07:40:09.814747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.814333 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn" event={"ID":"f1ab86b5-d305-436d-bb75-4430f3930c0d","Type":"ContainerDied","Data":"10efefe22be366b3a429c75723a9a3f1ee5a973c7bd1b30a42d20c1c944ae6b5"} Oct 30 07:40:09.814747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.814364 5894 scope.go:122] "RemoveContainer" containerID="56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b" Oct 30 07:40:09.825217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.825200 5894 scope.go:122] "RemoveContainer" containerID="56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b" Oct 30 07:40:09.825503 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:40:09.825483 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b\": container with ID starting with 56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b not found: ID does not exist" containerID="56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b" Oct 30 07:40:09.825558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.825512 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b"} err="failed to get container status \"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b\": rpc error: code = NotFound desc = could not find container \"56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b\": container with ID starting with 56ee0cb339c13d74eee1963ccad9708001024dc6b2c8e3264ab6de151e3f013b not found: ID does not exist" Oct 30 07:40:09.869237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.869206 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:40:09.880647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:09.880610 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5c7vvgn"] Oct 30 07:40:10.432667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:10.432593 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ab86b5-d305-436d-bb75-4430f3930c0d" path="/var/lib/kubelet/pods/f1ab86b5-d305-436d-bb75-4430f3930c0d/volumes" Oct 30 07:40:11.029657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:11.029606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:11.032803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:11.032779 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg"] Oct 30 07:40:11.045187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:11.045160 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02f5f5b5-f6bvg"] Oct 30 07:40:12.433238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:12.433192 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e41f78-c6f2-4e1b-bb3c-93712080dd92" path="/var/lib/kubelet/pods/52e41f78-c6f2-4e1b-bb3c-93712080dd92/volumes" Oct 30 07:40:12.799661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:12.799619 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:14.419969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:14.419918 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:14.538439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:14.538399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:16.220030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:16.219993 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:16.375754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:16.375721 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:40:17.954081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:17.954050 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:19.146993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:19.146961 5894 scope.go:122] "RemoveContainer" containerID="f606c5c94eca13a86721b4e36a3154365ae582a2d7e5b3e05bc82364497fee5e" Oct 30 07:40:19.721096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:19.721048 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:21.535176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:21.535140 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:21.720106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:21.720037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:21.855616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:21.855507 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:23.484434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:23.484401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:24.070268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.070232 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:24.326081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.325994 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:24.326799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.326777 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f1ab86b5-d305-436d-bb75-4430f3930c0d" containerName="payload-processing" Oct 30 07:40:24.326799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.326797 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ab86b5-d305-436d-bb75-4430f3930c0d" containerName="payload-processing" Oct 30 07:40:24.327010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.326990 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ab86b5-d305-436d-bb75-4430f3930c0d" containerName="payload-processing" Oct 30 07:40:24.333340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.333316 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.352795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.352765 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:24.361761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361721 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-e2e--f9d83428-dockercfg-kln78\"" Oct 30 07:40:24.361761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361748 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"agc-cred-d2c6e7565718\"" Oct 30 07:40:24.361761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361761 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"routing-overlay\"" Oct 30 07:40:24.362060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361878 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"kube-root-ca.crt\"" Oct 30 07:40:24.362060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361874 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"openshift-service-ca.crt\"" Oct 30 07:40:24.362060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.361999 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b096ec\"" Oct 30 07:40:24.399571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.399531 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhxj\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.399788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.399698 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.399788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.399719 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.399890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.399869 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.500986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.500946 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhxj\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.501500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.501143 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.501500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.501173 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.501500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.501356 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.501935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.501913 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.502172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.502152 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.503759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.503735 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.532200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.532161 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhxj\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.648588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.648498 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:24.957330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.957301 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:24.957580 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:40:24.957555 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3192b688_4e9e_4f73_b71e_085a7827af28.slice/crio-6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246 WatchSource:0}: Error finding container 6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246: Status 404 returned error can't find the container with id 6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246 Oct 30 07:40:24.975555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.975487 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" event={"ID":"3192b688-4e9e-4f73-b71e-085a7827af28","Type":"ContainerStarted","Data":"6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246"} Oct 30 07:40:24.994109 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.994071 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:40:25.000232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:24.999981 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.015070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.015040 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"agc-cred-2ee23c2876eb\"" Oct 30 07:40:25.020074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.020047 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:40:25.111319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.111291 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.111498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.111347 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.111498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.111487 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq962\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.111585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.111515 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.213317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.213219 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.213317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.213297 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.213673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.213447 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq962\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.213673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.213480 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.214081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.214052 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.214297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.214273 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.216488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.216466 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.233722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.233693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:25.237549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.237529 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq962\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.326625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.326592 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.634679 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:40:25.634615 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07be748_1557_4949_a9c2_866cf00d94b9.slice/crio-193b51583f29113c511046d3c4faa0c312ad81b7bddf7442bcb5d348654ce0b0 WatchSource:0}: Error finding container 193b51583f29113c511046d3c4faa0c312ad81b7bddf7442bcb5d348654ce0b0: Status 404 returned error can't find the container with id 193b51583f29113c511046d3c4faa0c312ad81b7bddf7442bcb5d348654ce0b0 Oct 30 07:40:25.635247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.635197 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:40:25.986706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.986571 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" event={"ID":"e07be748-1557-4949-a9c2-866cf00d94b9","Type":"ContainerStarted","Data":"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384"} Oct 30 07:40:25.986706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.986671 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" event={"ID":"e07be748-1557-4949-a9c2-866cf00d94b9","Type":"ContainerStarted","Data":"193b51583f29113c511046d3c4faa0c312ad81b7bddf7442bcb5d348654ce0b0"} Oct 30 07:40:25.986948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.986812 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:25.988403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.988376 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" event={"ID":"3192b688-4e9e-4f73-b71e-085a7827af28","Type":"ContainerStarted","Data":"aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095"} Oct 30 07:40:25.988536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:25.988517 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:26.029782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:26.029722 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" podStartSLOduration=2.029706687 podStartE2EDuration="2.029706687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:26.028837697 +0000 UTC m=+2589.719755135" watchObservedRunningTime="2026-10-30 07:40:26.029706687 +0000 UTC m=+2589.720624163" Oct 30 07:40:26.072541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:26.072486 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" podStartSLOduration=2.072472687 podStartE2EDuration="2.072472687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:26.070901712 +0000 UTC m=+2589.761819150" watchObservedRunningTime="2026-10-30 07:40:26.072472687 +0000 UTC m=+2589.763390125" Oct 30 07:40:26.919164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:26.919102 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:27.866366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.866332 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:40:27.871395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.871370 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.897473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.897431 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:40:27.952786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.952748 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.952867 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjt5\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.952944 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953098 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953154 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953207 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953304 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953355 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.953703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.953407 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:27.972608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:27.972570 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-705337-openshift-default-dockercfg-z72xk\"" Oct 30 07:40:28.055064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055025 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055088 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjt5\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055140 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055207 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055237 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055254 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055280 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055370 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055444 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055719 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055746 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055775 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.055858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.055818 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.056181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.056157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.057816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.057787 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.058130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.058112 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.082226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.082191 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.082880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.082854 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjt5\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.190060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.189976 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:28.527349 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:40:28.527318 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acb7928_044a_4865_b485_8d162cf734e3.slice/crio-6ec6ee8f37fe4c3838176bd6fd43dbb98a993156c65de218e22eb001e8fe5bb4 WatchSource:0}: Error finding container 6ec6ee8f37fe4c3838176bd6fd43dbb98a993156c65de218e22eb001e8fe5bb4: Status 404 returned error can't find the container with id 6ec6ee8f37fe4c3838176bd6fd43dbb98a993156c65de218e22eb001e8fe5bb4 Oct 30 07:40:28.528193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.528165 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:40:28.530189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.530156 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.530279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.530226 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.530279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.530259 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:28.657059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:28.657010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:29.015526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.015491 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" event={"ID":"9acb7928-044a-4865-b485-8d162cf734e3","Type":"ContainerStarted","Data":"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6"} Oct 30 07:40:29.015940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.015546 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" event={"ID":"9acb7928-044a-4865-b485-8d162cf734e3","Type":"ContainerStarted","Data":"6ec6ee8f37fe4c3838176bd6fd43dbb98a993156c65de218e22eb001e8fe5bb4"} Oct 30 07:40:29.074727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.074628 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podStartSLOduration=2.074613915 podStartE2EDuration="2.074613915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:29.073471932 +0000 UTC m=+2592.764389371" watchObservedRunningTime="2026-10-30 07:40:29.074613915 +0000 UTC m=+2592.765531352" Oct 30 07:40:29.190976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.190942 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:29.192711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.192676 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:29.192861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.192740 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:29.740687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.740650 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:29.849116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:29.849083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:30.126567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:30.126530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:30.193068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:30.193030 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:30.193355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:30.193327 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:30.245908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:30.245877 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:30.705381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:30.705345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:31.192108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:31.192071 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:31.192786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:31.192743 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:32.195244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:32.194752 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:32.195244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:32.194846 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:32.522883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:32.522772 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:33.197734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:33.197692 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:33.198228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:33.197747 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:34.170227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:34.170194 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:34.196077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:34.196034 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:34.196280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:34.196093 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:34.742187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:34.742137 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" podUID="3192b688-4e9e-4f73-b71e-085a7827af28" containerName="payload-processing" containerID="cri-o://aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095" gracePeriod=25 Oct 30 07:40:34.833399 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:40:34.833359 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:40:34.856027 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:40:34.855982 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3192b688_4e9e_4f73_b71e_085a7827af28.slice/crio-aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3192b688_4e9e_4f73_b71e_085a7827af28.slice/crio-conmon-aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:40:35.059594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.059564 5894 generic.go:356] "Generic (PLEG): container finished" podID="3192b688-4e9e-4f73-b71e-085a7827af28" containerID="aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095" exitCode=0 Oct 30 07:40:35.059818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.059666 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" event={"ID":"3192b688-4e9e-4f73-b71e-085a7827af28","Type":"ContainerDied","Data":"aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095"} Oct 30 07:40:35.059818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.059734 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" event={"ID":"3192b688-4e9e-4f73-b71e-085a7827af28","Type":"ContainerDied","Data":"6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246"} Oct 30 07:40:35.059818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.059758 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db2d1f7e8cefdccd3d7fcc4e25c5ece399c6adf1685ac00502219f0914ea246" Oct 30 07:40:35.068004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.067979 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:35.114020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.113983 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\") pod \"3192b688-4e9e-4f73-b71e-085a7827af28\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " Oct 30 07:40:35.114205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.114173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\") pod \"3192b688-4e9e-4f73-b71e-085a7827af28\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " Oct 30 07:40:35.114291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.114252 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\") pod \"3192b688-4e9e-4f73-b71e-085a7827af28\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " Oct 30 07:40:35.114372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.114355 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\") pod \"3192b688-4e9e-4f73-b71e-085a7827af28\" (UID: \"3192b688-4e9e-4f73-b71e-085a7827af28\") " Oct 30 07:40:35.115286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.115253 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume" pod "3192b688-4e9e-4f73-b71e-085a7827af28" (UID: "3192b688-4e9e-4f73-b71e-085a7827af28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:35.115614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.115595 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume" pod "3192b688-4e9e-4f73-b71e-085a7827af28" (UID: "3192b688-4e9e-4f73-b71e-085a7827af28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:35.126747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.126712 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj" pod "3192b688-4e9e-4f73-b71e-085a7827af28" (UID: "3192b688-4e9e-4f73-b71e-085a7827af28"). InnerVolumeSpecName "kube-api-access-7zhxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:35.126910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.126771 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials" pod "3192b688-4e9e-4f73-b71e-085a7827af28" (UID: "3192b688-4e9e-4f73-b71e-085a7827af28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:35.191247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.191214 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 07:40:35.191435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.191267 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 07:40:35.216438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.216389 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:35.216438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.216428 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhxj\" (UniqueName: \"kubernetes.io/projected/3192b688-4e9e-4f73-b71e-085a7827af28-kube-api-access-7zhxj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:35.216438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.216444 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:35.216889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.216458 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3192b688-4e9e-4f73-b71e-085a7827af28-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:35.980033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:35.979997 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:36.067434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.067403 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl" Oct 30 07:40:36.124257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.124225 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:36.136511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.136480 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-7bdchs9nl"] Oct 30 07:40:36.195580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.195546 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:36.196062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.196039 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:36.196735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.196719 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:40:36.438191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.438154 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3192b688-4e9e-4f73-b71e-085a7827af28" path="/var/lib/kubelet/pods/3192b688-4e9e-4f73-b71e-085a7827af28/volumes" Oct 30 07:40:36.999947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:36.999916 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:40:37.078912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:37.078879 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:40:37.736770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:37.736736 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:39.450219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:39.450174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:41.218963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:41.218924 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:42.080193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.080140 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" podUID="31f32bf0-fd7c-4ca3-a770-04236a20651f" containerName="payload-processing" containerID="cri-o://84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf" gracePeriod=25 Oct 30 07:40:42.353262 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.353234 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:40:42.435036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435000 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\") pod \"31f32bf0-fd7c-4ca3-a770-04236a20651f\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " Oct 30 07:40:42.435261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435130 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\") pod \"31f32bf0-fd7c-4ca3-a770-04236a20651f\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " Oct 30 07:40:42.435261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435183 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\") pod \"31f32bf0-fd7c-4ca3-a770-04236a20651f\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " Oct 30 07:40:42.435433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435414 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\") pod \"31f32bf0-fd7c-4ca3-a770-04236a20651f\" (UID: \"31f32bf0-fd7c-4ca3-a770-04236a20651f\") " Oct 30 07:40:42.435555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume" pod "31f32bf0-fd7c-4ca3-a770-04236a20651f" (UID: "31f32bf0-fd7c-4ca3-a770-04236a20651f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:42.435977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.435951 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:42.436370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.436342 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume" pod "31f32bf0-fd7c-4ca3-a770-04236a20651f" (UID: "31f32bf0-fd7c-4ca3-a770-04236a20651f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:40:42.438091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.438060 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st" pod "31f32bf0-fd7c-4ca3-a770-04236a20651f" (UID: "31f32bf0-fd7c-4ca3-a770-04236a20651f"). InnerVolumeSpecName "kube-api-access-rj7st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:42.438370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.438344 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials" pod "31f32bf0-fd7c-4ca3-a770-04236a20651f" (UID: "31f32bf0-fd7c-4ca3-a770-04236a20651f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:40:42.537505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.537456 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/31f32bf0-fd7c-4ca3-a770-04236a20651f-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:42.537505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.537488 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7st\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-kube-api-access-rj7st\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:42.537505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.537501 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/31f32bf0-fd7c-4ca3-a770-04236a20651f-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:40:42.812729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.812691 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:42.908440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.908393 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 07:40:42.909486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909459 5894 cpu_manager.go:414] "containerMap: removing container" podUID="31f32bf0-fd7c-4ca3-a770-04236a20651f" containerName="payload-processing" Oct 30 07:40:42.909486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909486 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f32bf0-fd7c-4ca3-a770-04236a20651f" containerName="payload-processing" Oct 30 07:40:42.909694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909575 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3192b688-4e9e-4f73-b71e-085a7827af28" containerName="payload-processing" Oct 30 07:40:42.909694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909583 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3192b688-4e9e-4f73-b71e-085a7827af28" containerName="payload-processing" Oct 30 07:40:42.909836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909821 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f32bf0-fd7c-4ca3-a770-04236a20651f" containerName="payload-processing" Oct 30 07:40:42.909898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.909856 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3192b688-4e9e-4f73-b71e-085a7827af28" containerName="payload-processing" Oct 30 07:40:42.958070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.958032 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 07:40:42.958278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:42.958182 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062428 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062616 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062697 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9hf\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062759 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062888 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.062960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062917 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.063018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.062956 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.063018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.063008 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.063127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.063042 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.137426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.137395 5894 generic.go:356] "Generic (PLEG): container finished" podID="31f32bf0-fd7c-4ca3-a770-04236a20651f" containerID="84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf" exitCode=0 Oct 30 07:40:43.137673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.137458 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" Oct 30 07:40:43.137673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.137481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" event={"ID":"31f32bf0-fd7c-4ca3-a770-04236a20651f","Type":"ContainerDied","Data":"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf"} Oct 30 07:40:43.137673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.137541 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq" event={"ID":"31f32bf0-fd7c-4ca3-a770-04236a20651f","Type":"ContainerDied","Data":"a9ead08c46102d6fb4baf25f6e17ae3e17c319725aaafe71f26728ad4c5eb292"} Oct 30 07:40:43.137673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.137569 5894 scope.go:122] "RemoveContainer" containerID="84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf" Oct 30 07:40:43.149375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.149348 5894 scope.go:122] "RemoveContainer" containerID="84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf" Oct 30 07:40:43.149698 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:40:43.149676 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf\": container with ID starting with 84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf not found: ID does not exist" containerID="84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf" Oct 30 07:40:43.149772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.149707 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf"} err="failed to get container status \"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf\": rpc error: code = NotFound desc = could not find container \"84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf\": container with ID starting with 84ffd37a810b7663eee62afbb81ea38a1580af9a8855169605fdcb4ad1e315bf not found: ID does not exist" Oct 30 07:40:43.164007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.163980 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164102 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164133 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164190 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164226 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164253 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164295 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164438 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164485 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9hf\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164534 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164485 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164627 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.164811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.164771 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.165162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.165136 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.166357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.166337 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.166734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.166715 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.188499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.188471 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.188905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.188883 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9hf\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.201062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.201031 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:40:43.204670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.204622 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c65wmsq"] Oct 30 07:40:43.276894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.276840 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:43.600818 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:40:43.600786 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda307f70f_8e15_40b8_b8b4_659c38b70b4c.slice/crio-f18e0aad5683628226153c007191c1df959974878fec7a4af3307f5f005ddb82 WatchSource:0}: Error finding container f18e0aad5683628226153c007191c1df959974878fec7a4af3307f5f005ddb82: Status 404 returned error can't find the container with id f18e0aad5683628226153c007191c1df959974878fec7a4af3307f5f005ddb82 Oct 30 07:40:43.601252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.600872 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 07:40:43.603877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.603844 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.603968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.603908 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:43.603968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:43.603936 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:44.147465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.147412 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" event={"ID":"a307f70f-8e15-40b8-b8b4-659c38b70b4c","Type":"ContainerStarted","Data":"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007"} Oct 30 07:40:44.147712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.147483 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" event={"ID":"a307f70f-8e15-40b8-b8b4-659c38b70b4c","Type":"ContainerStarted","Data":"f18e0aad5683628226153c007191c1df959974878fec7a4af3307f5f005ddb82"} Oct 30 07:40:44.206979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.206919 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podStartSLOduration=2.206903459 podStartE2EDuration="2.206903459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:44.205185304 +0000 UTC m=+2607.896102740" watchObservedRunningTime="2026-10-30 07:40:44.206903459 +0000 UTC m=+2607.897820958" Oct 30 07:40:44.277336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.277300 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:44.278705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.278672 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:44.278879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.278732 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:44.432611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.432530 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f32bf0-fd7c-4ca3-a770-04236a20651f" path="/var/lib/kubelet/pods/31f32bf0-fd7c-4ca3-a770-04236a20651f/volumes" Oct 30 07:40:44.644406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:44.644369 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:45.292659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:45.287076 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:45.292659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:45.287138 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:46.286119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:46.286079 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:46.286833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:46.286794 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:46.391659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:46.387240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:47.292660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:47.286666 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:47.292660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:47.286729 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:48.220276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:48.220236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:48.281249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:48.279071 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:48.281249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:48.279137 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:49.278079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:49.278041 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Oct 30 07:40:49.278713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:49.278677 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Oct 30 07:40:50.025384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:50.025347 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:50.282739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:50.282706 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:50.283256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:50.283076 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:50.283952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:50.283931 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 07:40:51.824368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:51.824333 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:53.429106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:53.429072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:55.135996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:55.135964 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:55.596803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:55.596769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:56.835847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:56.835763 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:57.946217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:57.946184 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:40:57.951964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:57.951925 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:57.974702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:57.974673 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:40:58.117297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117266 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvt28\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117300 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117389 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117487 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117588 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117622 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117707 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117733 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.117859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.117841 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219279 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219333 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvt28\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219356 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219405 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219453 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219589 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219661 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.219726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219699 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.220006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219782 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.220006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.219856 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.220466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.220442 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.220588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.220567 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.221000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.220980 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.222078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.222052 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.222586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.222557 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.243451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.243423 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvt28\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.246316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.246285 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\") pod \"e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.268249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.268220 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:40:58.428275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.428228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:40:58.591063 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:40:58.591030 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8e79fc_f42e_4162_9826_429e9181c65c.slice/crio-b7d5aeafbc88741ef86b5c3471ed5dddbac044ad7912ffcf1afd43dafbfe8fc7 WatchSource:0}: Error finding container b7d5aeafbc88741ef86b5c3471ed5dddbac044ad7912ffcf1afd43dafbfe8fc7: Status 404 returned error can't find the container with id b7d5aeafbc88741ef86b5c3471ed5dddbac044ad7912ffcf1afd43dafbfe8fc7 Oct 30 07:40:58.592164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.592123 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:40:58.593808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.593776 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:58.593905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.593838 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:58.593905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:58.593865 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:40:59.304545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:59.304504 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" event={"ID":"7b8e79fc-f42e-4162-9826-429e9181c65c","Type":"ContainerStarted","Data":"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43"} Oct 30 07:40:59.305010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:59.304610 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" event={"ID":"7b8e79fc-f42e-4162-9826-429e9181c65c","Type":"ContainerStarted","Data":"b7d5aeafbc88741ef86b5c3471ed5dddbac044ad7912ffcf1afd43dafbfe8fc7"} Oct 30 07:40:59.365732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:40:59.365680 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podStartSLOduration=2.365665422 podStartE2EDuration="2.365665422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:40:59.362826451 +0000 UTC m=+2623.053743887" watchObservedRunningTime="2026-10-30 07:40:59.365665422 +0000 UTC m=+2623.056582861" Oct 30 07:41:00.193269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:00.193233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:00.268811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:00.268778 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:41:00.270594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:00.270566 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 07:41:00.270797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:00.270616 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 07:41:01.268887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:01.268849 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 07:41:01.269314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:01.268909 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 07:41:02.087660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:02.086024 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:02.269488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:02.269441 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 07:41:02.269943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:02.269496 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 07:41:03.269700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:03.269657 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 07:41:03.270124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:03.269760 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 07:41:03.601337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:03.601282 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:03.946904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:03.946802 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:04.269328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:04.269236 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" start-of-body= Oct 30 07:41:04.269328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:04.269297 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.246:15021/healthz/ready\": dial tcp 10.134.0.246:15021: connect: connection refused" Oct 30 07:41:04.839342 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:41:04.839305 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:41:04.942887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:04.942856 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.091521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.091423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.218067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.218035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.273494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.273465 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:41:05.273916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.273893 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:41:05.274707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.274686 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:41:05.467723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.467615 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.620627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.620594 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.745313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.745212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:05.870843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:05.870812 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:06.267302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:06.267265 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:06.400742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:06.400708 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:07.603328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:07.603296 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:09.238690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:09.238619 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:10.828420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:10.828389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:12.467350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:12.467303 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:13.428665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:13.428581 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:13.542781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:13.542747 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:14.109484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:14.109447 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:41:14.175774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:14.175739 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:14.293432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:14.293399 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:15.940907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:15.940865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:17.034199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:17.034161 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw"] Oct 30 07:41:17.046845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:17.046810 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6b9ad7c-v6lbw"] Oct 30 07:41:17.618786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:17.618750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:18.283891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.283859 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx"] Oct 30 07:41:18.288950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.288925 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.308258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.308222 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx"] Oct 30 07:41:18.316789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.316681 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:41:18.316789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.316718 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:41:18.316789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.316713 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:41:18.316789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.316750 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:41:18.328506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.328475 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.328678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.328605 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5dbd\" (UniqueName: \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.328855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.328833 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.430216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.430184 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.430415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.430259 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.430415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.430356 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5dbd\" (UniqueName: \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.431219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.431192 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.432915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.432892 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.433997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.433976 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0afa885-a2aa-495e-b874-a28218e75e82" path="/var/lib/kubelet/pods/e0afa885-a2aa-495e-b874-a28218e75e82/volumes" Oct 30 07:41:18.454367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.454345 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5dbd\" (UniqueName: \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\") pod \"maas-api-revoke-sub-7b11c46e-pcnqx\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.604510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.604429 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:18.913534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:18.913503 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx"] Oct 30 07:41:18.913702 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:18.913549 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc9014_baee_420d_a6d5_df5ffcec87e7.slice/crio-4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63 WatchSource:0}: Error finding container 4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63: Status 404 returned error can't find the container with id 4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63 Oct 30 07:41:19.199172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.199087 5894 scope.go:122] "RemoveContainer" containerID="6cbb779978356e556716e85fdee02cc05d02613e7c0d08f6f1755be923459495" Oct 30 07:41:19.210337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.210302 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:19.215237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.215174 5894 scope.go:122] "RemoveContainer" containerID="0d78f8db5fe82d12081913b07fbdc8218dec554a281a1f41b31c1afb6f427ea2" Oct 30 07:41:19.227374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.227349 5894 scope.go:122] "RemoveContainer" containerID="5374d7b4b1f1ceb2090a6741ce08b80b2d0a2ba452930c4e2f75812129843d6d" Oct 30 07:41:19.239237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.239215 5894 scope.go:122] "RemoveContainer" containerID="8903627e6b1da5f2893a860c9d5635508cd5f6e3805e6f8f4d4e364a0b25b026" Oct 30 07:41:19.249709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.249686 5894 scope.go:122] "RemoveContainer" containerID="a9b7cfd14787e26e0c085746bf717d47b2daa48b29394c0c0de3ba50b36a01a6" Oct 30 07:41:19.259740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.259720 5894 scope.go:122] "RemoveContainer" containerID="360812004f295844d9cd12dc4d039d1ceca1341ced4ea69443dfafcf8a0f98cc" Oct 30 07:41:19.327182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.327159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:19.485722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.485590 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" event={"ID":"46bc9014-baee-420d-a6d5-df5ffcec87e7","Type":"ContainerStarted","Data":"2d4389edb580439e270241c3a22878e3f135c68b537e94da332b2e5842a42869"} Oct 30 07:41:19.485722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.485707 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" event={"ID":"46bc9014-baee-420d-a6d5-df5ffcec87e7","Type":"ContainerStarted","Data":"4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63"} Oct 30 07:41:19.554231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:19.554170 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" podStartSLOduration=1.5541491509999998 podStartE2EDuration="1.554149151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:19.541175199 +0000 UTC m=+2643.232092636" watchObservedRunningTime="2026-10-30 07:41:19.554149151 +0000 UTC m=+2643.245066589" Oct 30 07:41:20.161405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:20.161366 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:20.494384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:20.494298 5894 generic.go:356] "Generic (PLEG): container finished" podID="46bc9014-baee-420d-a6d5-df5ffcec87e7" containerID="2d4389edb580439e270241c3a22878e3f135c68b537e94da332b2e5842a42869" exitCode=0 Oct 30 07:41:20.494847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:20.494385 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" event={"ID":"46bc9014-baee-420d-a6d5-df5ffcec87e7","Type":"ContainerDied","Data":"2d4389edb580439e270241c3a22878e3f135c68b537e94da332b2e5842a42869"} Oct 30 07:41:21.042519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.042465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:21.168273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.168239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:21.283735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.283697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:21.412224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.412124 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:21.687774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.687744 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:21.795501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.795465 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\" (UniqueName: \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\") pod \"46bc9014-baee-420d-a6d5-df5ffcec87e7\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " Oct 30 07:41:21.795731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.795588 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\") pod \"46bc9014-baee-420d-a6d5-df5ffcec87e7\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " Oct 30 07:41:21.795820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.795749 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\") pod \"46bc9014-baee-420d-a6d5-df5ffcec87e7\" (UID: \"46bc9014-baee-420d-a6d5-df5ffcec87e7\") " Oct 30 07:41:21.796041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.796005 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca" pod "46bc9014-baee-420d-a6d5-df5ffcec87e7" (UID: "46bc9014-baee-420d-a6d5-df5ffcec87e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:21.796358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.796337 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:21.797766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.797737 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd" pod "46bc9014-baee-420d-a6d5-df5ffcec87e7" (UID: "46bc9014-baee-420d-a6d5-df5ffcec87e7"). InnerVolumeSpecName "kube-api-access-t5dbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:21.797870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.797786 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca" pod "46bc9014-baee-420d-a6d5-df5ffcec87e7" (UID: "46bc9014-baee-420d-a6d5-df5ffcec87e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:41:21.898212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.898147 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5dbd\" (UniqueName: \"kubernetes.io/projected/46bc9014-baee-420d-a6d5-df5ffcec87e7-kube-api-access-t5dbd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:21.898212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:21.898210 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46bc9014-baee-420d-a6d5-df5ffcec87e7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:22.510902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:22.510877 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" Oct 30 07:41:22.511083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:22.510907 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx" event={"ID":"46bc9014-baee-420d-a6d5-df5ffcec87e7","Type":"ContainerDied","Data":"4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63"} Oct 30 07:41:22.511083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:22.510954 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d38d91a5d8bf6b78bd8383e4b65d90505f306191d4c14c291d713569da11f63" Oct 30 07:41:23.042932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:23.042896 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:24.770723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:24.770693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:25.204141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.204104 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn"] Oct 30 07:41:25.205301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.205282 5894 cpu_manager.go:414] "containerMap: removing container" podUID="46bc9014-baee-420d-a6d5-df5ffcec87e7" containerName="revoke-keys" Oct 30 07:41:25.205373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.205305 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46bc9014-baee-420d-a6d5-df5ffcec87e7" containerName="revoke-keys" Oct 30 07:41:25.205508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.205497 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="46bc9014-baee-420d-a6d5-df5ffcec87e7" containerName="revoke-keys" Oct 30 07:41:25.209864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.209838 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.228613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.228579 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn"] Oct 30 07:41:25.236222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.236182 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:41:25.236371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.236227 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:41:25.239602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.239578 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:41:25.239812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.239759 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:41:25.347360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.347322 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvr4\" (UniqueName: \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.347548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.347424 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.347548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.347464 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.448500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.448470 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvr4\" (UniqueName: \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.448752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.448515 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.448752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.448548 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.449191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.449168 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.451083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.451053 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.472508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.472442 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvr4\" (UniqueName: \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\") pod \"maas-api-revoke-sub-876ec31b-x68sn\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.530105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.530075 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:25.839093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:25.839063 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn"] Oct 30 07:41:25.839573 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:25.839526 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda4c275_529d_424e_8b75_ac6e910437fb.slice/crio-7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b WatchSource:0}: Error finding container 7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b: Status 404 returned error can't find the container with id 7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b Oct 30 07:41:26.376714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.376569 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:26.502532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.502497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:26.543144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.543101 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" event={"ID":"cda4c275-529d-424e-8b75-ac6e910437fb","Type":"ContainerStarted","Data":"7b46a94d71cf2c8f5ea2a4281a0482426fd0fd739a5b8a340fb719efdd855ccf"} Oct 30 07:41:26.543334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.543163 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" event={"ID":"cda4c275-529d-424e-8b75-ac6e910437fb","Type":"ContainerStarted","Data":"7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b"} Oct 30 07:41:26.587974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.587600 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" podStartSLOduration=1.587581842 podStartE2EDuration="1.587581842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:26.584157212 +0000 UTC m=+2650.275074649" watchObservedRunningTime="2026-10-30 07:41:26.587581842 +0000 UTC m=+2650.278499279" Oct 30 07:41:26.627713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:26.625922 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:27.551620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:27.551587 5894 generic.go:356] "Generic (PLEG): container finished" podID="cda4c275-529d-424e-8b75-ac6e910437fb" containerID="7b46a94d71cf2c8f5ea2a4281a0482426fd0fd739a5b8a340fb719efdd855ccf" exitCode=0 Oct 30 07:41:27.552133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:27.551675 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" event={"ID":"cda4c275-529d-424e-8b75-ac6e910437fb","Type":"ContainerDied","Data":"7b46a94d71cf2c8f5ea2a4281a0482426fd0fd739a5b8a340fb719efdd855ccf"} Oct 30 07:41:28.032053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.032005 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc"] Oct 30 07:41:28.035125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.035091 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-086f927b-x7ltc"] Oct 30 07:41:28.293986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.293893 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.416884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.416843 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.436466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.436433 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8491a39-dda0-487b-b2d6-6833422828ec" path="/var/lib/kubelet/pods/c8491a39-dda0-487b-b2d6-6833422828ec/volumes" Oct 30 07:41:28.536576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.536533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.666544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.666501 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.758904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.758875 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:28.789055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.789024 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.883964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.883125 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\") pod \"cda4c275-529d-424e-8b75-ac6e910437fb\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " Oct 30 07:41:28.883964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.883235 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\" (UniqueName: \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\") pod \"cda4c275-529d-424e-8b75-ac6e910437fb\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " Oct 30 07:41:28.883964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.883338 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\") pod \"cda4c275-529d-424e-8b75-ac6e910437fb\" (UID: \"cda4c275-529d-424e-8b75-ac6e910437fb\") " Oct 30 07:41:28.883964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.883574 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca" pod "cda4c275-529d-424e-8b75-ac6e910437fb" (UID: "cda4c275-529d-424e-8b75-ac6e910437fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:28.884338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.884102 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:28.886555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.886533 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca" pod "cda4c275-529d-424e-8b75-ac6e910437fb" (UID: "cda4c275-529d-424e-8b75-ac6e910437fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:41:28.886777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.886751 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4" pod "cda4c275-529d-424e-8b75-ac6e910437fb" (UID: "cda4c275-529d-424e-8b75-ac6e910437fb"). InnerVolumeSpecName "kube-api-access-fdvr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:28.912265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.912217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:28.985222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.985177 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdvr4\" (UniqueName: \"kubernetes.io/projected/cda4c275-529d-424e-8b75-ac6e910437fb-kube-api-access-fdvr4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:28.985222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:28.985216 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cda4c275-529d-424e-8b75-ac6e910437fb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:29.034757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.034717 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:29.231759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.231483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:29.569671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.569629 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" Oct 30 07:41:29.569854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.569629 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn" event={"ID":"cda4c275-529d-424e-8b75-ac6e910437fb","Type":"ContainerDied","Data":"7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b"} Oct 30 07:41:29.569902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.569870 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a6c2432731597ea1a39256490d00411a7b0871318b527deab0f8b8b672b5b0b" Oct 30 07:41:29.637835 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.637787 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:29.775225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.775190 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:29.849676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.849557 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:41:29.945167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:29.945054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.063684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.063629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.187671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.187528 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.313812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.312423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.623513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.621184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.744659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.742003 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:30.978665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:30.978545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:31.143891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.143855 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:31.211316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.211276 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq"] Oct 30 07:41:31.212475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.212449 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cda4c275-529d-424e-8b75-ac6e910437fb" containerName="revoke-keys" Oct 30 07:41:31.212596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.212479 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda4c275-529d-424e-8b75-ac6e910437fb" containerName="revoke-keys" Oct 30 07:41:31.212677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.212623 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda4c275-529d-424e-8b75-ac6e910437fb" containerName="revoke-keys" Oct 30 07:41:31.216589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.216571 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.231628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.231543 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.231798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.231764 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtt4k\" (UniqueName: \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.231957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.231934 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.234472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.234444 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq"] Oct 30 07:41:31.245168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.244395 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:41:31.245168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.244480 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:41:31.245168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.244534 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:41:31.245168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.244605 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:41:31.265397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.265354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:31.359659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.335688 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtt4k\" (UniqueName: \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.380985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.360171 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.381402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.381365 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.399666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.382521 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.401266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.401236 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtt4k\" (UniqueName: \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.428498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.428464 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b20aac82-s2jhq\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.434606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.434571 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:31.500854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.500776 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:31.505894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.505872 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.525357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.525331 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"agc-cred-090d55727a49\"" Oct 30 07:41:31.525610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.525591 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"payload-processing-external-model-e2e--0aef5a31-dockercfg-29vwr\"" Oct 30 07:41:31.526294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.526270 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:31.531499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.531472 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"openshift-service-ca.crt\"" Oct 30 07:41:31.531723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.531701 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"kube-root-ca.crt\"" Oct 30 07:41:31.534980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.534955 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6051b2\"" Oct 30 07:41:31.535281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.535223 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"routing-overlay\"" Oct 30 07:41:31.536433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.536409 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:31.583831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.583801 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.584028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.583858 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.584028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.583930 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pd8\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.584028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.583991 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.717656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.705975 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.717656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.706064 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.717656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.706100 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pd8\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.717656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.706130 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.717656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.707291 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.729868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.729802 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.737660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.734541 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.767349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.767267 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pd8\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.825500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.825462 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:31.863103 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:31.863071 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862dfeb2_98bb_4b60_bbda_bb33805957bb.slice/crio-8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59 WatchSource:0}: Error finding container 8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59: Status 404 returned error can't find the container with id 8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59 Oct 30 07:41:31.866302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:31.866270 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq"] Oct 30 07:41:32.091663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.086277 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:32.147931 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:32.147899 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ae7bfa_57ec_4f9f_937b_bffe2cc43678.slice/crio-f821e5c254fa5bae056c4decd1efcde3af08b446154025e7829939ec476561f0 WatchSource:0}: Error finding container f821e5c254fa5bae056c4decd1efcde3af08b446154025e7829939ec476561f0: Status 404 returned error can't find the container with id f821e5c254fa5bae056c4decd1efcde3af08b446154025e7829939ec476561f0 Oct 30 07:41:32.148824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.148797 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:32.204428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.204397 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:32.614877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.614843 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" event={"ID":"76ae7bfa-57ec-4f9f-937b-bffe2cc43678","Type":"ContainerStarted","Data":"46e722f3d2c1996b788586fe8e5cb86bc6323c91688b3135a6512adc00dbafb6"} Oct 30 07:41:32.615084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.614913 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:32.615084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.614936 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" event={"ID":"76ae7bfa-57ec-4f9f-937b-bffe2cc43678","Type":"ContainerStarted","Data":"f821e5c254fa5bae056c4decd1efcde3af08b446154025e7829939ec476561f0"} Oct 30 07:41:32.616730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.616697 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" event={"ID":"862dfeb2-98bb-4b60-bbda-bb33805957bb","Type":"ContainerStarted","Data":"f2ffc4a034d81959b1d890540c4d3d940cdfd4565354c652802ff3991f966a29"} Oct 30 07:41:32.616855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.616747 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" event={"ID":"862dfeb2-98bb-4b60-bbda-bb33805957bb","Type":"ContainerStarted","Data":"8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59"} Oct 30 07:41:32.658588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.658525 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" podStartSLOduration=1.6585114349999999 podStartE2EDuration="1.658511435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:32.656135572 +0000 UTC m=+2656.347053006" watchObservedRunningTime="2026-10-30 07:41:32.658511435 +0000 UTC m=+2656.349428869" Oct 30 07:41:32.704009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.703953 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" podStartSLOduration=1.703938929 podStartE2EDuration="1.703938929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:32.69958494 +0000 UTC m=+2656.390502379" watchObservedRunningTime="2026-10-30 07:41:32.703938929 +0000 UTC m=+2656.394856369" Oct 30 07:41:32.840286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:32.840254 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:33.625966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:33.625930 5894 generic.go:356] "Generic (PLEG): container finished" podID="862dfeb2-98bb-4b60-bbda-bb33805957bb" containerID="f2ffc4a034d81959b1d890540c4d3d940cdfd4565354c652802ff3991f966a29" exitCode=0 Oct 30 07:41:33.626482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:33.626015 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" event={"ID":"862dfeb2-98bb-4b60-bbda-bb33805957bb","Type":"ContainerDied","Data":"f2ffc4a034d81959b1d890540c4d3d940cdfd4565354c652802ff3991f966a29"} Oct 30 07:41:34.652163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:34.652129 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:34.799300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:34.799261 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:34.837568 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:41:34.837528 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:41:34.850589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:34.850554 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" podUID="0236897a-10f8-43fe-bb88-3cd1f26de668" containerName="payload-processing" containerID="cri-o://85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d" gracePeriod=25 Oct 30 07:41:34.878418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:34.878390 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:35.020657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:34.994551 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\") pod \"862dfeb2-98bb-4b60-bbda-bb33805957bb\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " Oct 30 07:41:35.020657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.013974 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca" pod "862dfeb2-98bb-4b60-bbda-bb33805957bb" (UID: "862dfeb2-98bb-4b60-bbda-bb33805957bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:35.020657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.015832 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\") pod \"862dfeb2-98bb-4b60-bbda-bb33805957bb\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " Oct 30 07:41:35.020657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.016079 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\" (UniqueName: \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\") pod \"862dfeb2-98bb-4b60-bbda-bb33805957bb\" (UID: \"862dfeb2-98bb-4b60-bbda-bb33805957bb\") " Oct 30 07:41:35.020657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.016944 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.044191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.036245 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k" pod "862dfeb2-98bb-4b60-bbda-bb33805957bb" (UID: "862dfeb2-98bb-4b60-bbda-bb33805957bb"). InnerVolumeSpecName "kube-api-access-rtt4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:35.044191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.043000 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca" pod "862dfeb2-98bb-4b60-bbda-bb33805957bb" (UID: "862dfeb2-98bb-4b60-bbda-bb33805957bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:41:35.119129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.118902 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtt4k\" (UniqueName: \"kubernetes.io/projected/862dfeb2-98bb-4b60-bbda-bb33805957bb-kube-api-access-rtt4k\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.119129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.118944 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/862dfeb2-98bb-4b60-bbda-bb33805957bb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.218233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.218203 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:41:35.346719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.342751 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume" pod "0236897a-10f8-43fe-bb88-3cd1f26de668" (UID: "0236897a-10f8-43fe-bb88-3cd1f26de668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:35.346992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.346966 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\") pod \"0236897a-10f8-43fe-bb88-3cd1f26de668\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " Oct 30 07:41:35.371664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.357851 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\") pod \"0236897a-10f8-43fe-bb88-3cd1f26de668\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " Oct 30 07:41:35.371664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.358247 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\") pod \"0236897a-10f8-43fe-bb88-3cd1f26de668\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " Oct 30 07:41:35.371664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.358285 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\") pod \"0236897a-10f8-43fe-bb88-3cd1f26de668\" (UID: \"0236897a-10f8-43fe-bb88-3cd1f26de668\") " Oct 30 07:41:35.371664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.358875 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.371664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.360153 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume" pod "0236897a-10f8-43fe-bb88-3cd1f26de668" (UID: "0236897a-10f8-43fe-bb88-3cd1f26de668"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:35.381679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.377803 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials" pod "0236897a-10f8-43fe-bb88-3cd1f26de668" (UID: "0236897a-10f8-43fe-bb88-3cd1f26de668"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:35.386131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.384380 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9" pod "0236897a-10f8-43fe-bb88-3cd1f26de668" (UID: "0236897a-10f8-43fe-bb88-3cd1f26de668"). InnerVolumeSpecName "kube-api-access-xg9p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:35.460482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.460386 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0236897a-10f8-43fe-bb88-3cd1f26de668-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.460482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.460425 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.460482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.460439 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg9p9\" (UniqueName: \"kubernetes.io/projected/0236897a-10f8-43fe-bb88-3cd1f26de668-kube-api-access-xg9p9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:35.646267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.646236 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" Oct 30 07:41:35.646267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.646250 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq" event={"ID":"862dfeb2-98bb-4b60-bbda-bb33805957bb","Type":"ContainerDied","Data":"8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59"} Oct 30 07:41:35.646577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.646308 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d678b6c35c69e8766d86dabde6963fe8eea4929472be1cc7ee6ddc17a6edd59" Oct 30 07:41:35.648155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.648128 5894 generic.go:356] "Generic (PLEG): container finished" podID="0236897a-10f8-43fe-bb88-3cd1f26de668" containerID="85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d" exitCode=0 Oct 30 07:41:35.648289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.648173 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" Oct 30 07:41:35.648289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.648198 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" event={"ID":"0236897a-10f8-43fe-bb88-3cd1f26de668","Type":"ContainerDied","Data":"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d"} Oct 30 07:41:35.648289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.648250 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d" event={"ID":"0236897a-10f8-43fe-bb88-3cd1f26de668","Type":"ContainerDied","Data":"aec2a48e7bae75a9cbdbacf880515ab863b3b0ecea1da8a7419fd4e7873a9db1"} Oct 30 07:41:35.648289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.648276 5894 scope.go:122] "RemoveContainer" containerID="85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d" Oct 30 07:41:35.664865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.664828 5894 scope.go:122] "RemoveContainer" containerID="85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d" Oct 30 07:41:35.665286 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:41:35.665261 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d\": container with ID starting with 85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d not found: ID does not exist" containerID="85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d" Oct 30 07:41:35.665334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.665298 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d"} err="failed to get container status \"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d\": rpc error: code = NotFound desc = could not find container \"85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d\": container with ID starting with 85ba62827d25fcb1325513686fcaea75bc2d46fee7ccd089ccee1128709c746d not found: ID does not exist" Oct 30 07:41:35.718557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.718484 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:41:35.725199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:35.725168 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-59dfzxn9d"] Oct 30 07:41:36.057412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.057375 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:36.482509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.482415 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0236897a-10f8-43fe-bb88-3cd1f26de668" path="/var/lib/kubelet/pods/0236897a-10f8-43fe-bb88-3cd1f26de668/volumes" Oct 30 07:41:36.969226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.969195 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:41:36.969943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.969927 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0236897a-10f8-43fe-bb88-3cd1f26de668" containerName="payload-processing" Oct 30 07:41:36.969999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.969945 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0236897a-10f8-43fe-bb88-3cd1f26de668" containerName="payload-processing" Oct 30 07:41:36.969999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.969983 5894 cpu_manager.go:414] "containerMap: removing container" podUID="862dfeb2-98bb-4b60-bbda-bb33805957bb" containerName="revoke-keys" Oct 30 07:41:36.969999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.969990 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="862dfeb2-98bb-4b60-bbda-bb33805957bb" containerName="revoke-keys" Oct 30 07:41:36.970110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.970101 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0236897a-10f8-43fe-bb88-3cd1f26de668" containerName="payload-processing" Oct 30 07:41:36.970148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.970118 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="862dfeb2-98bb-4b60-bbda-bb33805957bb" containerName="revoke-keys" Oct 30 07:41:36.975426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.975404 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.013660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:36.996852 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:41:37.020688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.020627 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6051b2\"/\"agc-cred-b34c1803f8b5\"" Oct 30 07:41:37.117124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.117089 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.117380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.117360 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.117452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.117413 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhq97\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.117517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.117496 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.218833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.218800 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.219057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.218849 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhq97\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.219057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.218884 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.219057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.218961 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.219718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.219622 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.219801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.219733 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.221792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.221765 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.243071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.243042 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhq97\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.290251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.290210 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:37.356087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.356035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:37.605622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.604305 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:41:37.645117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.645074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:37.762140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.762105 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:37.835740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:37.835709 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:41:38.086881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.086842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.204572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.204494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.344158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.344127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.485894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.485801 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.605088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.605053 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.677966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.677930 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" event={"ID":"69f122cc-8751-4b60-be36-ce4514ad9051","Type":"ContainerStarted","Data":"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0"} Oct 30 07:41:38.678154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.677987 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" event={"ID":"69f122cc-8751-4b60-be36-ce4514ad9051","Type":"ContainerStarted","Data":"d5738db2be45aa2a12fd8dff8215c28ff6937a482b017044b1b129f803b5fb95"} Oct 30 07:41:38.678154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.678050 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:38.738133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.737617 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" podStartSLOduration=2.737595631 podStartE2EDuration="2.737595631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:38.734790526 +0000 UTC m=+2662.425707964" watchObservedRunningTime="2026-10-30 07:41:38.737595631 +0000 UTC m=+2662.428513068" Oct 30 07:41:38.777278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.777240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:38.901705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:38.901673 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:39.399805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:39.399766 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:39.536359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:39.536316 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:40.520222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:40.520185 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:40.633580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:40.633543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:40.748199 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:40.748162 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:41.331168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.331131 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:41.646887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.646803 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:41:41.651460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.651441 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.672598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.672570 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:41:41.680250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.680225 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"payload-processing-external-model-e2e--cc0b775b-dockercfg-btmr5\"" Oct 30 07:41:41.680476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.680224 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"agc-cred-9f181eb40533\"" Oct 30 07:41:41.680476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.680334 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1e0880\"/\"agc-cred-2a7a67b3a81f\"" Oct 30 07:41:41.835187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.832807 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.836029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.835999 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.836442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.836423 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.836647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.836610 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7blv\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.938498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.938403 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.938498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.938485 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7blv\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.938772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.938711 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.938841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.938798 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.939429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.939407 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.939518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.939469 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.941315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.941287 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.962366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.962330 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7blv\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:41.967206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:41.967177 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:42.278486 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:42.278451 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274c887e_5217_455d_a409_2b4c7473a08c.slice/crio-7efcdf16e67a1d7c4366ee4dc86e862bb9245c9d4d6ab6fe72dc4f5ea8516f95 WatchSource:0}: Error finding container 7efcdf16e67a1d7c4366ee4dc86e862bb9245c9d4d6ab6fe72dc4f5ea8516f95: Status 404 returned error can't find the container with id 7efcdf16e67a1d7c4366ee4dc86e862bb9245c9d4d6ab6fe72dc4f5ea8516f95 Oct 30 07:41:42.281045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.281016 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:41:42.605983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.605940 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" podUID="24b75f30-f708-41d9-8ef9-0f57db93ef67" containerName="payload-processing" containerID="cri-o://ec20c0af3d9227266313f18fb33989aa8d5243884a8f09c6db58edb53867d851" gracePeriod=25 Oct 30 07:41:42.805846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.805798 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" event={"ID":"274c887e-5217-455d-a409-2b4c7473a08c","Type":"ContainerStarted","Data":"32b80457c905aeafdb12a58556b5bcd377ecc192b2b2520b24f7715338d06f48"} Oct 30 07:41:42.806350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.805866 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" event={"ID":"274c887e-5217-455d-a409-2b4c7473a08c","Type":"ContainerStarted","Data":"7efcdf16e67a1d7c4366ee4dc86e862bb9245c9d4d6ab6fe72dc4f5ea8516f95"} Oct 30 07:41:42.806350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.805984 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:42.808813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.808789 5894 generic.go:356] "Generic (PLEG): container finished" podID="24b75f30-f708-41d9-8ef9-0f57db93ef67" containerID="ec20c0af3d9227266313f18fb33989aa8d5243884a8f09c6db58edb53867d851" exitCode=0 Oct 30 07:41:42.808956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.808821 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" event={"ID":"24b75f30-f708-41d9-8ef9-0f57db93ef67","Type":"ContainerDied","Data":"ec20c0af3d9227266313f18fb33989aa8d5243884a8f09c6db58edb53867d851"} Oct 30 07:41:42.817268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.817240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:42.871075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.870729 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" podStartSLOduration=1.8707071499999999 podStartE2EDuration="1.87070715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:41:42.848479653 +0000 UTC m=+2666.539397089" watchObservedRunningTime="2026-10-30 07:41:42.87070715 +0000 UTC m=+2666.561624588" Oct 30 07:41:42.903050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:42.903007 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:41:43.029735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.029689 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:41:43.044266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.044230 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:43.075137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.075098 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\") pod \"24b75f30-f708-41d9-8ef9-0f57db93ef67\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " Oct 30 07:41:43.075349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.075154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\") pod \"24b75f30-f708-41d9-8ef9-0f57db93ef67\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " Oct 30 07:41:43.075349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.075221 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\") pod \"24b75f30-f708-41d9-8ef9-0f57db93ef67\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " Oct 30 07:41:43.075349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.075276 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\") pod \"24b75f30-f708-41d9-8ef9-0f57db93ef67\" (UID: \"24b75f30-f708-41d9-8ef9-0f57db93ef67\") " Oct 30 07:41:43.075942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.075915 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume" pod "24b75f30-f708-41d9-8ef9-0f57db93ef67" (UID: "24b75f30-f708-41d9-8ef9-0f57db93ef67"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:43.076532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.076506 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume" pod "24b75f30-f708-41d9-8ef9-0f57db93ef67" (UID: "24b75f30-f708-41d9-8ef9-0f57db93ef67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:43.079149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.079123 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz" pod "24b75f30-f708-41d9-8ef9-0f57db93ef67" (UID: "24b75f30-f708-41d9-8ef9-0f57db93ef67"). InnerVolumeSpecName "kube-api-access-rpzjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:43.079651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.079616 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials" pod "24b75f30-f708-41d9-8ef9-0f57db93ef67" (UID: "24b75f30-f708-41d9-8ef9-0f57db93ef67"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:43.158872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.158778 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:43.177926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.177885 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:43.177926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.177920 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/24b75f30-f708-41d9-8ef9-0f57db93ef67-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:43.177926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.177932 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzjz\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-kube-api-access-rpzjz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:43.178173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.177944 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/24b75f30-f708-41d9-8ef9-0f57db93ef67-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:43.630215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.630185 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:43.761126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.761091 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:43.817706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.817662 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" event={"ID":"24b75f30-f708-41d9-8ef9-0f57db93ef67","Type":"ContainerDied","Data":"62f4f7cf901b084a6e50bb6af3d86ddf4508f4afdd84315f5e54c1fce7826f7b"} Oct 30 07:41:43.818219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.817745 5894 scope.go:122] "RemoveContainer" containerID="ec20c0af3d9227266313f18fb33989aa8d5243884a8f09c6db58edb53867d851" Oct 30 07:41:43.818219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.817680 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm" Oct 30 07:41:43.872846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.872813 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:41:43.877204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:43.877174 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-76dbxzpkm"] Oct 30 07:41:44.433269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:44.433233 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b75f30-f708-41d9-8ef9-0f57db93ef67" path="/var/lib/kubelet/pods/24b75f30-f708-41d9-8ef9-0f57db93ef67/volumes" Oct 30 07:41:44.540805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:44.540774 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:46.217007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:46.216970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:47.789312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:47.789279 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:48.031663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.031603 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" podUID="161ab4fa-123c-4b7e-826e-9c8587bb003c" containerName="payload-processing" containerID="cri-o://fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b" gracePeriod=25 Oct 30 07:41:48.320887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.320859 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:41:48.362453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.362371 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\") pod \"161ab4fa-123c-4b7e-826e-9c8587bb003c\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " Oct 30 07:41:48.363519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.363491 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\") pod \"161ab4fa-123c-4b7e-826e-9c8587bb003c\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " Oct 30 07:41:48.363936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.363910 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\") pod \"161ab4fa-123c-4b7e-826e-9c8587bb003c\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " Oct 30 07:41:48.364173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.364106 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\") pod \"161ab4fa-123c-4b7e-826e-9c8587bb003c\" (UID: \"161ab4fa-123c-4b7e-826e-9c8587bb003c\") " Oct 30 07:41:48.365242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.365187 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume" pod "161ab4fa-123c-4b7e-826e-9c8587bb003c" (UID: "161ab4fa-123c-4b7e-826e-9c8587bb003c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:48.365592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.365572 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume" pod "161ab4fa-123c-4b7e-826e-9c8587bb003c" (UID: "161ab4fa-123c-4b7e-826e-9c8587bb003c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:48.366394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.366371 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:48.366515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.366398 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/161ab4fa-123c-4b7e-826e-9c8587bb003c-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:48.378206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.367219 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials" pod "161ab4fa-123c-4b7e-826e-9c8587bb003c" (UID: "161ab4fa-123c-4b7e-826e-9c8587bb003c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:48.386356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.386320 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p" pod "161ab4fa-123c-4b7e-826e-9c8587bb003c" (UID: "161ab4fa-123c-4b7e-826e-9c8587bb003c"). InnerVolumeSpecName "kube-api-access-lpz9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:48.468134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.468094 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpz9p\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-kube-api-access-lpz9p\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:48.468134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.468126 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/161ab4fa-123c-4b7e-826e-9c8587bb003c-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:48.875458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.875424 5894 generic.go:356] "Generic (PLEG): container finished" podID="161ab4fa-123c-4b7e-826e-9c8587bb003c" containerID="fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b" exitCode=0 Oct 30 07:41:48.875893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.875485 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" Oct 30 07:41:48.875893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.875502 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" event={"ID":"161ab4fa-123c-4b7e-826e-9c8587bb003c","Type":"ContainerDied","Data":"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b"} Oct 30 07:41:48.875893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.875556 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr" event={"ID":"161ab4fa-123c-4b7e-826e-9c8587bb003c","Type":"ContainerDied","Data":"05dd83d8ea4d948f1968eba7687070832d1b2f3682ccc12b40353cba5a3929c8"} Oct 30 07:41:48.875893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.875579 5894 scope.go:122] "RemoveContainer" containerID="fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b" Oct 30 07:41:48.886056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.886018 5894 scope.go:122] "RemoveContainer" containerID="fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b" Oct 30 07:41:48.886366 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:41:48.886344 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b\": container with ID starting with fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b not found: ID does not exist" containerID="fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b" Oct 30 07:41:48.886448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.886374 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b"} err="failed to get container status \"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b\": rpc error: code = NotFound desc = could not find container \"fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b\": container with ID starting with fa934b057109f69a3f134411de929c381c40ce2b67a2310ae4dc1b86f61a9c3b not found: ID does not exist" Oct 30 07:41:48.933778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.933739 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:41:48.945917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:48.945880 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-69ddwh9rr"] Oct 30 07:41:49.366405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:49.366359 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:49.719399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:49.719317 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:41:49.800044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:49.800004 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:50.239038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:50.239004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:50.366960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:50.366926 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:50.433412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:50.433377 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161ab4fa-123c-4b7e-826e-9c8587bb003c" path="/var/lib/kubelet/pods/161ab4fa-123c-4b7e-826e-9c8587bb003c/volumes" Oct 30 07:41:50.484901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:50.484868 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:51.041092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:51.041057 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:52.831563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:52.831524 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:53.822099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:53.822072 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:41:54.556601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:54.556565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:54.800937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:54.800895 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" podUID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" containerName="payload-processing" containerID="cri-o://46e722f3d2c1996b788586fe8e5cb86bc6323c91688b3135a6512adc00dbafb6" gracePeriod=25 Oct 30 07:41:54.930398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:54.930360 5894 generic.go:356] "Generic (PLEG): container finished" podID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" containerID="46e722f3d2c1996b788586fe8e5cb86bc6323c91688b3135a6512adc00dbafb6" exitCode=0 Oct 30 07:41:54.930542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:54.930391 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" event={"ID":"76ae7bfa-57ec-4f9f-937b-bffe2cc43678","Type":"ContainerDied","Data":"46e722f3d2c1996b788586fe8e5cb86bc6323c91688b3135a6512adc00dbafb6"} Oct 30 07:41:55.086415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.086347 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.197976 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\") pod \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.198194 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\") pod \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.198329 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\") pod \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.198437 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\") pod \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\" (UID: \"76ae7bfa-57ec-4f9f-937b-bffe2cc43678\") " Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.198753 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume" pod "76ae7bfa-57ec-4f9f-937b-bffe2cc43678" (UID: "76ae7bfa-57ec-4f9f-937b-bffe2cc43678"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:55.199347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.199347 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:55.199833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.199460 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume" pod "76ae7bfa-57ec-4f9f-937b-bffe2cc43678" (UID: "76ae7bfa-57ec-4f9f-937b-bffe2cc43678"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:41:55.201942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.201915 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8" pod "76ae7bfa-57ec-4f9f-937b-bffe2cc43678" (UID: "76ae7bfa-57ec-4f9f-937b-bffe2cc43678"). InnerVolumeSpecName "kube-api-access-k8pd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:55.201942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.201933 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials" pod "76ae7bfa-57ec-4f9f-937b-bffe2cc43678" (UID: "76ae7bfa-57ec-4f9f-937b-bffe2cc43678"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:41:55.300561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.300527 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:55.300561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.300559 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:55.300561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.300570 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pd8\" (UniqueName: \"kubernetes.io/projected/76ae7bfa-57ec-4f9f-937b-bffe2cc43678-kube-api-access-k8pd8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:41:55.940089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.940062 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" Oct 30 07:41:55.940575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.940062 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv" event={"ID":"76ae7bfa-57ec-4f9f-937b-bffe2cc43678","Type":"ContainerDied","Data":"f821e5c254fa5bae056c4decd1efcde3af08b446154025e7829939ec476561f0"} Oct 30 07:41:55.940575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.940207 5894 scope.go:122] "RemoveContainer" containerID="46e722f3d2c1996b788586fe8e5cb86bc6323c91688b3135a6512adc00dbafb6" Oct 30 07:41:55.998446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:55.998406 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:56.010225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:56.010189 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bvc6rv"] Oct 30 07:41:56.155717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:56.155682 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:56.435339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:56.435307 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" path="/var/lib/kubelet/pods/76ae7bfa-57ec-4f9f-937b-bffe2cc43678/volumes" Oct 30 07:41:56.485391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:56.485349 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:56.616217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:56.616183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:57.466774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:57.466733 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:57.588206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:57.588168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:58.108805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:58.108765 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:58.877339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:58.877305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:41:59.129962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.129880 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:41:59.130665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130649 5894 cpu_manager.go:414] "containerMap: removing container" podUID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" containerName="payload-processing" Oct 30 07:41:59.130728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130666 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" containerName="payload-processing" Oct 30 07:41:59.130728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130692 5894 cpu_manager.go:414] "containerMap: removing container" podUID="161ab4fa-123c-4b7e-826e-9c8587bb003c" containerName="payload-processing" Oct 30 07:41:59.130728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130698 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161ab4fa-123c-4b7e-826e-9c8587bb003c" containerName="payload-processing" Oct 30 07:41:59.130728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130708 5894 cpu_manager.go:414] "containerMap: removing container" podUID="24b75f30-f708-41d9-8ef9-0f57db93ef67" containerName="payload-processing" Oct 30 07:41:59.130728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130714 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b75f30-f708-41d9-8ef9-0f57db93ef67" containerName="payload-processing" Oct 30 07:41:59.130894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130810 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="161ab4fa-123c-4b7e-826e-9c8587bb003c" containerName="payload-processing" Oct 30 07:41:59.130894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130820 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b75f30-f708-41d9-8ef9-0f57db93ef67" containerName="payload-processing" Oct 30 07:41:59.130894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.130835 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ae7bfa-57ec-4f9f-937b-bffe2cc43678" containerName="payload-processing" Oct 30 07:41:59.135669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.135643 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.155423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.155375 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:41:59.164059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164024 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 07:41:59.164059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164047 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 07:41:59.164059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164054 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 07:41:59.164375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164201 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2hz7h\"" Oct 30 07:41:59.164375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164264 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 07:41:59.164375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.164311 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d36827793baf\"" Oct 30 07:41:59.182422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.182384 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.182627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.182462 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.182627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.182534 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.182793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.182767 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnmm\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.308668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.290842 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnmm\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.386753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.381836 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.386753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.381932 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.386753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.381960 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.394661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.394059 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.395664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.394909 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.395664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.372429 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnmm\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.408958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.408038 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\") pod \"payload-processing-external-model-847555f7c-47r6d\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.450985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.450947 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:41:59.764518 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:41:59.764481 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4563d70a_a3d5_41ee_b299_0f0fe1c19554.slice/crio-9db123c3812023f2d6c7c34c10627c2833692310b35fc83bcd27436f779b959a WatchSource:0}: Error finding container 9db123c3812023f2d6c7c34c10627c2833692310b35fc83bcd27436f779b959a: Status 404 returned error can't find the container with id 9db123c3812023f2d6c7c34c10627c2833692310b35fc83bcd27436f779b959a Oct 30 07:41:59.765455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.765428 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:41:59.917687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:41:59.917606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:00.009867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:00.009827 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" event={"ID":"4563d70a-a3d5-41ee-b299-0f0fe1c19554","Type":"ContainerStarted","Data":"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba"} Oct 30 07:42:00.010071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:00.009898 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" event={"ID":"4563d70a-a3d5-41ee-b299-0f0fe1c19554","Type":"ContainerStarted","Data":"9db123c3812023f2d6c7c34c10627c2833692310b35fc83bcd27436f779b959a"} Oct 30 07:42:00.010071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:00.009930 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:42:00.053788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:00.053732 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" podStartSLOduration=1.053719514 podStartE2EDuration="1.053719514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:00.051813876 +0000 UTC m=+2683.742731313" watchObservedRunningTime="2026-10-30 07:42:00.053719514 +0000 UTC m=+2683.744636948" Oct 30 07:42:01.712997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:01.712963 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:03.546231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:03.546194 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:04.831991 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:42:04.831945 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:42:05.225127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:05.225043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:06.813538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:06.813509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:06.904749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:06.904702 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:42:08.883400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:08.883361 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:10.607148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:10.607113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:11.021786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:11.021697 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:42:12.390970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.390928 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx"] Oct 30 07:42:12.397071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.397040 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.419011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.418951 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:12.422844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.422818 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:42:12.423517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.423491 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx"] Oct 30 07:42:12.426781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.426758 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:42:12.426962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.426758 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:42:12.427068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.426788 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:42:12.436862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.436830 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxvl\" (UniqueName: \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.437229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.437200 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.437522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.437485 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.552659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.543574 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.552659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.543677 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxvl\" (UniqueName: \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.552659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.543834 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.558775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.558253 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.560955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.560925 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.581654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.578732 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxvl\" (UniqueName: \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\") pod \"maas-api-revoke-sub-03c6453b-cphfx\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:12.716579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:12.716503 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:13.026489 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:13.026449 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd885cdeb_aef2_4231_947a_960e8f40639b.slice/crio-b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756 WatchSource:0}: Error finding container b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756: Status 404 returned error can't find the container with id b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756 Oct 30 07:42:13.027993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:13.027953 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx"] Oct 30 07:42:13.142359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:13.142205 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" event={"ID":"d885cdeb-aef2-4231-947a-960e8f40639b","Type":"ContainerStarted","Data":"b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756"} Oct 30 07:42:14.064419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:14.064385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:14.154181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:14.154082 5894 generic.go:356] "Generic (PLEG): container finished" podID="d885cdeb-aef2-4231-947a-960e8f40639b" containerID="8ac907ac5589759f235bc1d75bbf0fd63a78cbf334b7f8f5c4a29e888327854b" exitCode=0 Oct 30 07:42:14.154181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:14.154163 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" event={"ID":"d885cdeb-aef2-4231-947a-960e8f40639b","Type":"ContainerDied","Data":"8ac907ac5589759f235bc1d75bbf0fd63a78cbf334b7f8f5c4a29e888327854b"} Oct 30 07:42:15.330814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.330789 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:15.384350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.384318 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\") pod \"d885cdeb-aef2-4231-947a-960e8f40639b\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " Oct 30 07:42:15.384559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.384420 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\" (UniqueName: \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\") pod \"d885cdeb-aef2-4231-947a-960e8f40639b\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " Oct 30 07:42:15.384676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.384544 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\") pod \"d885cdeb-aef2-4231-947a-960e8f40639b\" (UID: \"d885cdeb-aef2-4231-947a-960e8f40639b\") " Oct 30 07:42:15.385796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.384819 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca" pod "d885cdeb-aef2-4231-947a-960e8f40639b" (UID: "d885cdeb-aef2-4231-947a-960e8f40639b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:15.385796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.385317 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:15.387193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.387162 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca" pod "d885cdeb-aef2-4231-947a-960e8f40639b" (UID: "d885cdeb-aef2-4231-947a-960e8f40639b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:42:15.387883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.387855 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl" pod "d885cdeb-aef2-4231-947a-960e8f40639b" (UID: "d885cdeb-aef2-4231-947a-960e8f40639b"). InnerVolumeSpecName "kube-api-access-6kxvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:15.486790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.486599 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxvl\" (UniqueName: \"kubernetes.io/projected/d885cdeb-aef2-4231-947a-960e8f40639b-kube-api-access-6kxvl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:15.487276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.486730 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d885cdeb-aef2-4231-947a-960e8f40639b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:15.738046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:15.737940 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:16.170785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:16.170754 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" Oct 30 07:42:16.170964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:16.170784 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx" event={"ID":"d885cdeb-aef2-4231-947a-960e8f40639b","Type":"ContainerDied","Data":"b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756"} Oct 30 07:42:16.170964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:16.170864 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9945dbae50762aa1106a40be9772196e23f555dd33a0081ef8027946afe3756" Oct 30 07:42:17.325068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:17.325020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:19.035073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:19.035041 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:19.334391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:19.334299 5894 scope.go:122] "RemoveContainer" containerID="99ecaf77acfeae7e2f8d4ac0ca86ca90870d2c668170d198cb20f1cac2457b5f" Oct 30 07:42:20.627571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:20.627534 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:22.285670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:22.285603 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:23.779529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:23.779491 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:23.919585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:23.919550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:24.133101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.133020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:24.276169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.276101 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:24.276956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.276935 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d885cdeb-aef2-4231-947a-960e8f40639b" containerName="revoke-keys" Oct 30 07:42:24.277028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.276958 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d885cdeb-aef2-4231-947a-960e8f40639b" containerName="revoke-keys" Oct 30 07:42:24.277104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.277094 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d885cdeb-aef2-4231-947a-960e8f40639b" containerName="revoke-keys" Oct 30 07:42:24.282519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.282496 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.301981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.301950 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:24.427545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.427460 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.427545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.427514 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.427545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.427536 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkgt6\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.427880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.427686 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.436697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.436662 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn"] Oct 30 07:42:24.446160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.446126 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.453496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.453464 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn"] Oct 30 07:42:24.462805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.462771 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:42:24.462961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.462851 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:42:24.462961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.462953 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:42:24.463164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.463144 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.555481 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.555535 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkgt6\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.555655 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.555858 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.555959 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.556078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.558661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.556111 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq98n\" (UniqueName: \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.570661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.570449 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.574656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.571025 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.599215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.598343 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.622658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.615004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkgt6\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:24.657490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.657444 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.657713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.657523 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.657713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.657600 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq98n\" (UniqueName: \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.658757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.658735 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.661475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.661442 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.682146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.682077 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq98n\" (UniqueName: \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\") pod \"maas-api-revoke-sub-843d03e2-flxmn\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.768376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.768343 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:24.897291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:24.897249 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:25.074530 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:25.074489 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c19ed8_c212_4d43_b14f_80adba57dbcf.slice/crio-e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735 WatchSource:0}: Error finding container e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735: Status 404 returned error can't find the container with id e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735 Oct 30 07:42:25.075280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:25.075255 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn"] Oct 30 07:42:25.207677 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:25.207596 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dc81d4_5700_4582_a724_93af4e414224.slice/crio-62ffea9cb477791723b13a52719b8e16042e297cbe650f10375470d2a9714777 WatchSource:0}: Error finding container 62ffea9cb477791723b13a52719b8e16042e297cbe650f10375470d2a9714777: Status 404 returned error can't find the container with id 62ffea9cb477791723b13a52719b8e16042e297cbe650f10375470d2a9714777 Oct 30 07:42:25.208144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:25.208120 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:25.238154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:25.238116 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" event={"ID":"10dc81d4-5700-4582-a724-93af4e414224","Type":"ContainerStarted","Data":"62ffea9cb477791723b13a52719b8e16042e297cbe650f10375470d2a9714777"} Oct 30 07:42:25.240074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:25.240041 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" event={"ID":"73c19ed8-c212-4d43-b14f-80adba57dbcf","Type":"ContainerStarted","Data":"e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735"} Oct 30 07:42:25.842369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:25.842335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:26.250122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:26.250027 5894 generic.go:356] "Generic (PLEG): container finished" podID="73c19ed8-c212-4d43-b14f-80adba57dbcf" containerID="2246ab4f183a14ef000fc00189537e08f981bf45238d67843aa38433d9861332" exitCode=0 Oct 30 07:42:26.250573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:26.250114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" event={"ID":"73c19ed8-c212-4d43-b14f-80adba57dbcf","Type":"ContainerDied","Data":"2246ab4f183a14ef000fc00189537e08f981bf45238d67843aa38433d9861332"} Oct 30 07:42:26.252014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:26.251978 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" event={"ID":"10dc81d4-5700-4582-a724-93af4e414224","Type":"ContainerStarted","Data":"2cd84e6c19d1efcd5aecbb5afd867e14a7dd02c52417b587d2252c3abc10093d"} Oct 30 07:42:26.252147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:26.252074 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:26.338151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:26.338097 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" podStartSLOduration=2.338081618 podStartE2EDuration="2.338081618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:26.334560446 +0000 UTC m=+2710.025477881" watchObservedRunningTime="2026-10-30 07:42:26.338081618 +0000 UTC m=+2710.028999093" Oct 30 07:42:27.436500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.436471 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:27.502564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.502526 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\") pod \"73c19ed8-c212-4d43-b14f-80adba57dbcf\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " Oct 30 07:42:27.502817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.502795 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\") pod \"73c19ed8-c212-4d43-b14f-80adba57dbcf\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " Oct 30 07:42:27.502993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.502976 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\" (UniqueName: \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\") pod \"73c19ed8-c212-4d43-b14f-80adba57dbcf\" (UID: \"73c19ed8-c212-4d43-b14f-80adba57dbcf\") " Oct 30 07:42:27.503555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.503487 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca" pod "73c19ed8-c212-4d43-b14f-80adba57dbcf" (UID: "73c19ed8-c212-4d43-b14f-80adba57dbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:27.505753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.505725 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n" pod "73c19ed8-c212-4d43-b14f-80adba57dbcf" (UID: "73c19ed8-c212-4d43-b14f-80adba57dbcf"). InnerVolumeSpecName "kube-api-access-qq98n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:27.506139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.506114 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca" pod "73c19ed8-c212-4d43-b14f-80adba57dbcf" (UID: "73c19ed8-c212-4d43-b14f-80adba57dbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:42:27.515723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.515543 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:27.604822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.604787 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:27.604822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.604817 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq98n\" (UniqueName: \"kubernetes.io/projected/73c19ed8-c212-4d43-b14f-80adba57dbcf-kube-api-access-qq98n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:27.604822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:27.604826 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c19ed8-c212-4d43-b14f-80adba57dbcf-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:28.269919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:28.269873 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" event={"ID":"73c19ed8-c212-4d43-b14f-80adba57dbcf","Type":"ContainerDied","Data":"e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735"} Oct 30 07:42:28.270138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:28.269930 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e819aecb5c15b122527585b32e503063c72482f99d1776b3e1809e87ffc7f735" Oct 30 07:42:28.270138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:28.269889 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn" Oct 30 07:42:29.117979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:29.117936 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:30.402168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.402130 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:30.755369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.755284 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:30.892863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.892834 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl"] Oct 30 07:42:30.893645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.893616 5894 cpu_manager.go:414] "containerMap: removing container" podUID="73c19ed8-c212-4d43-b14f-80adba57dbcf" containerName="revoke-keys" Oct 30 07:42:30.893707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.893652 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c19ed8-c212-4d43-b14f-80adba57dbcf" containerName="revoke-keys" Oct 30 07:42:30.893795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.893785 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c19ed8-c212-4d43-b14f-80adba57dbcf" containerName="revoke-keys" Oct 30 07:42:30.898075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.898055 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:30.917940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.917905 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl"] Oct 30 07:42:30.925208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.925175 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:42:30.925208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.925213 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:42:30.925489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.925216 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:42:30.925979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.925959 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:42:30.962972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.962940 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:30.963145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.963115 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:30.963190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:30.963167 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4kbl\" (UniqueName: \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.065200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.065165 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.065392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.065260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4kbl\" (UniqueName: \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.065392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.065379 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.065963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.065933 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.067843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.067819 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.089956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.089922 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4kbl\" (UniqueName: \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\") pod \"maas-api-revoke-sub-8992c559-n4vfl\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.213380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.213348 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:31.524750 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:31.524705 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574bad66_f295_4324_842f_19f180c7e65d.slice/crio-a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c WatchSource:0}: Error finding container a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c: Status 404 returned error can't find the container with id a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c Oct 30 07:42:31.525559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:31.525533 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl"] Oct 30 07:42:32.312823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:32.312783 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" event={"ID":"574bad66-f295-4324-842f-19f180c7e65d","Type":"ContainerStarted","Data":"43dd825126ce0ad9f7a5fed9f39f71d4b92dc31432df122d6e358e43a195b228"} Oct 30 07:42:32.313011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:32.312841 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" event={"ID":"574bad66-f295-4324-842f-19f180c7e65d","Type":"ContainerStarted","Data":"a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c"} Oct 30 07:42:32.356044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:32.355991 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" podStartSLOduration=2.355976606 podStartE2EDuration="2.355976606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:32.353577805 +0000 UTC m=+2716.044495242" watchObservedRunningTime="2026-10-30 07:42:32.355976606 +0000 UTC m=+2716.046894044" Oct 30 07:42:32.538121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:32.538079 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:33.322887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:33.322847 5894 generic.go:356] "Generic (PLEG): container finished" podID="574bad66-f295-4324-842f-19f180c7e65d" containerID="43dd825126ce0ad9f7a5fed9f39f71d4b92dc31432df122d6e358e43a195b228" exitCode=0 Oct 30 07:42:33.323145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:33.322935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" event={"ID":"574bad66-f295-4324-842f-19f180c7e65d","Type":"ContainerDied","Data":"43dd825126ce0ad9f7a5fed9f39f71d4b92dc31432df122d6e358e43a195b228"} Oct 30 07:42:34.120985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.120950 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:34.516948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.516918 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:34.615497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.615459 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\") pod \"574bad66-f295-4324-842f-19f180c7e65d\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " Oct 30 07:42:34.615747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.615523 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\") pod \"574bad66-f295-4324-842f-19f180c7e65d\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " Oct 30 07:42:34.615850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.615819 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\" (UniqueName: \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\") pod \"574bad66-f295-4324-842f-19f180c7e65d\" (UID: \"574bad66-f295-4324-842f-19f180c7e65d\") " Oct 30 07:42:34.615944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.615915 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca" pod "574bad66-f295-4324-842f-19f180c7e65d" (UID: "574bad66-f295-4324-842f-19f180c7e65d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:34.616303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.616283 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/574bad66-f295-4324-842f-19f180c7e65d-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:34.617887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.617855 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca" pod "574bad66-f295-4324-842f-19f180c7e65d" (UID: "574bad66-f295-4324-842f-19f180c7e65d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:42:34.618250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.618232 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl" pod "574bad66-f295-4324-842f-19f180c7e65d" (UID: "574bad66-f295-4324-842f-19f180c7e65d"). InnerVolumeSpecName "kube-api-access-m4kbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:34.717248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.717162 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4kbl\" (UniqueName: \"kubernetes.io/projected/574bad66-f295-4324-842f-19f180c7e65d-kube-api-access-m4kbl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:34.717248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:34.717196 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/574bad66-f295-4324-842f-19f180c7e65d-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:34.829583 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:42:34.829540 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:42:35.339925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:35.339891 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" Oct 30 07:42:35.340336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:35.339927 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl" event={"ID":"574bad66-f295-4324-842f-19f180c7e65d","Type":"ContainerDied","Data":"a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c"} Oct 30 07:42:35.340336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:35.339977 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a042c757a97bad97acf7427eccc2b03e68ca9a3590a7c799edb3ebcb5ddd168c" Oct 30 07:42:35.730299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:35.730215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:35.890533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:35.890483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:36.011738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.011624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:36.173188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.173153 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:36.176260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.176235 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:36.209277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.209241 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:36.209987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.209972 5894 cpu_manager.go:414] "containerMap: removing container" podUID="574bad66-f295-4324-842f-19f180c7e65d" containerName="revoke-keys" Oct 30 07:42:36.210053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.209988 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574bad66-f295-4324-842f-19f180c7e65d" containerName="revoke-keys" Oct 30 07:42:36.210137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.210127 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="574bad66-f295-4324-842f-19f180c7e65d" containerName="revoke-keys" Oct 30 07:42:36.213909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.213889 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.232831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.232796 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:36.350860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.350821 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.351268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.350933 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kndqq\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.351268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.351018 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.351268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.351124 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.363706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.363671 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd"] Oct 30 07:42:36.409677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.409625 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd"] Oct 30 07:42:36.409990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.409974 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.431653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.431605 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:42:36.432608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.432582 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:42:36.432876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.432860 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:42:36.433058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.433044 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:42:36.452729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.452686 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.454143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.454116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kndqq\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.454856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.454833 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.455541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.455523 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.478979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.456382 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.480593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.454004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.493388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.493353 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.524356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.524323 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kndqq\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.534135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.534102 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:36.580674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.579731 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.580674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.579777 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.580674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.579847 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjldb\" (UniqueName: \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.682074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.682030 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.682296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.682087 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.682296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.682150 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjldb\" (UniqueName: \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.683432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.683398 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.685570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.685544 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.706224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.706185 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjldb\" (UniqueName: \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\") pod \"maas-api-revoke-sub-0c0b593a-t88rd\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.784452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.784416 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:36.872041 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:36.871929 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4a0931_f108_4971_8708_325cb2feacb9.slice/crio-287befcd1542f604c0de8601a73726e4b96179d29c108c75f37ea6cf44dfedbd WatchSource:0}: Error finding container 287befcd1542f604c0de8601a73726e4b96179d29c108c75f37ea6cf44dfedbd: Status 404 returned error can't find the container with id 287befcd1542f604c0de8601a73726e4b96179d29c108c75f37ea6cf44dfedbd Oct 30 07:42:36.873011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:36.872985 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:37.098674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.098623 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd"] Oct 30 07:42:37.098855 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:37.098811 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4251461_cf8f_4484_87c3_fff9d3c3dce6.slice/crio-35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea WatchSource:0}: Error finding container 35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea: Status 404 returned error can't find the container with id 35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea Oct 30 07:42:37.356876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.356831 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" event={"ID":"ba4a0931-f108-4971-8708-325cb2feacb9","Type":"ContainerStarted","Data":"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c"} Oct 30 07:42:37.357351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.356909 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:37.357351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.356935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" event={"ID":"ba4a0931-f108-4971-8708-325cb2feacb9","Type":"ContainerStarted","Data":"287befcd1542f604c0de8601a73726e4b96179d29c108c75f37ea6cf44dfedbd"} Oct 30 07:42:37.358379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.358329 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" event={"ID":"f4251461-cf8f-4484-87c3-fff9d3c3dce6","Type":"ContainerStarted","Data":"35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea"} Oct 30 07:42:37.399581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.399475 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" podStartSLOduration=1.399461794 podStartE2EDuration="1.399461794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:37.398184665 +0000 UTC m=+2721.089102103" watchObservedRunningTime="2026-10-30 07:42:37.399461794 +0000 UTC m=+2721.090379231" Oct 30 07:42:37.714557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:37.714474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:38.367395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:38.367301 5894 generic.go:356] "Generic (PLEG): container finished" podID="f4251461-cf8f-4484-87c3-fff9d3c3dce6" containerID="41ab1dc87e086ef86d88c833ba6b2773cd161b584e5fc017be66a54ebdbf8126" exitCode=0 Oct 30 07:42:38.367930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:38.367387 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" event={"ID":"f4251461-cf8f-4484-87c3-fff9d3c3dce6","Type":"ContainerDied","Data":"41ab1dc87e086ef86d88c833ba6b2773cd161b584e5fc017be66a54ebdbf8126"} Oct 30 07:42:39.432823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.432788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:39.591091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.591062 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:39.732493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.732403 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\") pod \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " Oct 30 07:42:39.732493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.732444 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\") pod \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " Oct 30 07:42:39.732749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.732499 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\" (UniqueName: \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\") pod \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\" (UID: \"f4251461-cf8f-4484-87c3-fff9d3c3dce6\") " Oct 30 07:42:39.732874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.732850 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca" pod "f4251461-cf8f-4484-87c3-fff9d3c3dce6" (UID: "f4251461-cf8f-4484-87c3-fff9d3c3dce6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:39.733200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.733183 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:39.734690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.734657 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca" pod "f4251461-cf8f-4484-87c3-fff9d3c3dce6" (UID: "f4251461-cf8f-4484-87c3-fff9d3c3dce6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:42:39.734804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.734752 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb" pod "f4251461-cf8f-4484-87c3-fff9d3c3dce6" (UID: "f4251461-cf8f-4484-87c3-fff9d3c3dce6"). InnerVolumeSpecName "kube-api-access-vjldb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:39.834486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.834441 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4251461-cf8f-4484-87c3-fff9d3c3dce6-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:39.834486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:39.834477 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjldb\" (UniqueName: \"kubernetes.io/projected/f4251461-cf8f-4484-87c3-fff9d3c3dce6-kube-api-access-vjldb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:40.385507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:40.385470 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" Oct 30 07:42:40.385707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:40.385469 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd" event={"ID":"f4251461-cf8f-4484-87c3-fff9d3c3dce6","Type":"ContainerDied","Data":"35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea"} Oct 30 07:42:40.385707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:40.385604 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35583f933e75e45e00420e6727910757074e087b92aa8c5ee1bf9359ec688fea" Oct 30 07:42:41.125288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.125253 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:41.190660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.175717 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" podUID="10dc81d4-5700-4582-a724-93af4e414224" containerName="payload-processing" containerID="cri-o://2cd84e6c19d1efcd5aecbb5afd867e14a7dd02c52417b587d2252c3abc10093d" gracePeriod=25 Oct 30 07:42:41.400129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.400032 5894 generic.go:356] "Generic (PLEG): container finished" podID="10dc81d4-5700-4582-a724-93af4e414224" containerID="2cd84e6c19d1efcd5aecbb5afd867e14a7dd02c52417b587d2252c3abc10093d" exitCode=0 Oct 30 07:42:41.400129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.400112 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" event={"ID":"10dc81d4-5700-4582-a724-93af4e414224","Type":"ContainerDied","Data":"2cd84e6c19d1efcd5aecbb5afd867e14a7dd02c52417b587d2252c3abc10093d"} Oct 30 07:42:41.475106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.475079 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:41.559381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.559348 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\") pod \"10dc81d4-5700-4582-a724-93af4e414224\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " Oct 30 07:42:41.559583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.559398 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\") pod \"10dc81d4-5700-4582-a724-93af4e414224\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " Oct 30 07:42:41.559583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.559458 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\") pod \"10dc81d4-5700-4582-a724-93af4e414224\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " Oct 30 07:42:41.559583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.559482 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\") pod \"10dc81d4-5700-4582-a724-93af4e414224\" (UID: \"10dc81d4-5700-4582-a724-93af4e414224\") " Oct 30 07:42:41.559886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.559857 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume" pod "10dc81d4-5700-4582-a724-93af4e414224" (UID: "10dc81d4-5700-4582-a724-93af4e414224"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:41.560035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.560007 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume" pod "10dc81d4-5700-4582-a724-93af4e414224" (UID: "10dc81d4-5700-4582-a724-93af4e414224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:41.560191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.560171 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:41.560254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.560196 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10dc81d4-5700-4582-a724-93af4e414224-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:41.561723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.561690 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6" pod "10dc81d4-5700-4582-a724-93af4e414224" (UID: "10dc81d4-5700-4582-a724-93af4e414224"). InnerVolumeSpecName "kube-api-access-rkgt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:41.561723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.561704 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials" pod "10dc81d4-5700-4582-a724-93af4e414224" (UID: "10dc81d4-5700-4582-a724-93af4e414224"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:41.661149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.661055 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkgt6\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-kube-api-access-rkgt6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:41.661149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:41.661093 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/10dc81d4-5700-4582-a724-93af4e414224-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:42.047337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.047299 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:42:42.149146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.149106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:42.259410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.259369 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:42.465718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.453447 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" Oct 30 07:42:42.465718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.464318 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w" event={"ID":"10dc81d4-5700-4582-a724-93af4e414224","Type":"ContainerDied","Data":"62ffea9cb477791723b13a52719b8e16042e297cbe650f10375470d2a9714777"} Oct 30 07:42:42.465718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.464391 5894 scope.go:122] "RemoveContainer" containerID="2cd84e6c19d1efcd5aecbb5afd867e14a7dd02c52417b587d2252c3abc10093d" Oct 30 07:42:42.542723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.542684 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:42.546078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:42.546046 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-54c6g897w"] Oct 30 07:42:43.027906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:43.027851 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:44.324818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:44.324095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:44.435173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:44.435131 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dc81d4-5700-4582-a724-93af4e414224" path="/var/lib/kubelet/pods/10dc81d4-5700-4582-a724-93af4e414224/volumes" Oct 30 07:42:44.466514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:44.466483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:45.247613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:45.247565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:45.799615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:45.799582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:46.047132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.047092 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:46.048006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.047981 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f4251461-cf8f-4484-87c3-fff9d3c3dce6" containerName="revoke-keys" Oct 30 07:42:46.048006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.048005 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4251461-cf8f-4484-87c3-fff9d3c3dce6" containerName="revoke-keys" Oct 30 07:42:46.048203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.048051 5894 cpu_manager.go:414] "containerMap: removing container" podUID="10dc81d4-5700-4582-a724-93af4e414224" containerName="payload-processing" Oct 30 07:42:46.048203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.048060 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dc81d4-5700-4582-a724-93af4e414224" containerName="payload-processing" Oct 30 07:42:46.048312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.048241 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4251461-cf8f-4484-87c3-fff9d3c3dce6" containerName="revoke-keys" Oct 30 07:42:46.048312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.048266 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dc81d4-5700-4582-a724-93af4e414224" containerName="payload-processing" Oct 30 07:42:46.052563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.052500 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.076101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.076066 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:46.080806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.080766 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-r5s6w\"" Oct 30 07:42:46.081420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.081397 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9b79a44aa3d6\"" Oct 30 07:42:46.155518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.155478 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.155755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.155587 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.155755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.155710 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.155885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.155761 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5458\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.265924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.265305 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.265924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.265392 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5458\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.265924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.265563 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.265924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.265670 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.267192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.266748 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.269008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.268596 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.290755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.282899 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.293992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.293961 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5458\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\") pod \"payload-processing-external-model-864476c44c-xtvlx\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.368419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.368315 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:46.691309 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:46.691276 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06c5120_8e56_46a7_b3fb_0e1f8fab88bf.slice/crio-98a77a5afc31ca53c11460553347734a83b07a232220eb13783faf55f052b775 WatchSource:0}: Error finding container 98a77a5afc31ca53c11460553347734a83b07a232220eb13783faf55f052b775: Status 404 returned error can't find the container with id 98a77a5afc31ca53c11460553347734a83b07a232220eb13783faf55f052b775 Oct 30 07:42:46.692885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:46.692805 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:47.047876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.047830 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" podUID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" containerName="payload-processing" containerID="cri-o://6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba" gracePeriod=25 Oct 30 07:42:47.052158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.052122 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:47.336903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.336874 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:42:47.397027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.396993 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\") pod \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " Oct 30 07:42:47.397361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.397048 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\") pod \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " Oct 30 07:42:47.397497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.397472 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume" pod "4563d70a-a3d5-41ee-b299-0f0fe1c19554" (UID: "4563d70a-a3d5-41ee-b299-0f0fe1c19554"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:47.397561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.397497 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume" pod "4563d70a-a3d5-41ee-b299-0f0fe1c19554" (UID: "4563d70a-a3d5-41ee-b299-0f0fe1c19554"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:47.397796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.397768 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\") pod \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " Oct 30 07:42:47.397917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.397826 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\") pod \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\" (UID: \"4563d70a-a3d5-41ee-b299-0f0fe1c19554\") " Oct 30 07:42:47.398818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.398497 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:47.398818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.398521 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4563d70a-a3d5-41ee-b299-0f0fe1c19554-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:47.402066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.402039 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials" pod "4563d70a-a3d5-41ee-b299-0f0fe1c19554" (UID: "4563d70a-a3d5-41ee-b299-0f0fe1c19554"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:47.402162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.402134 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm" pod "4563d70a-a3d5-41ee-b299-0f0fe1c19554" (UID: "4563d70a-a3d5-41ee-b299-0f0fe1c19554"). InnerVolumeSpecName "kube-api-access-dmnmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:47.500287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.500250 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:47.500287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.500284 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnmm\" (UniqueName: \"kubernetes.io/projected/4563d70a-a3d5-41ee-b299-0f0fe1c19554-kube-api-access-dmnmm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:47.513789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.513751 5894 generic.go:356] "Generic (PLEG): container finished" podID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" containerID="6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba" exitCode=0 Oct 30 07:42:47.513991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.513791 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" event={"ID":"4563d70a-a3d5-41ee-b299-0f0fe1c19554","Type":"ContainerDied","Data":"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba"} Oct 30 07:42:47.513991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.513867 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" event={"ID":"4563d70a-a3d5-41ee-b299-0f0fe1c19554","Type":"ContainerDied","Data":"9db123c3812023f2d6c7c34c10627c2833692310b35fc83bcd27436f779b959a"} Oct 30 07:42:47.513991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.513803 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-847555f7c-47r6d" Oct 30 07:42:47.513991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.513898 5894 scope.go:122] "RemoveContainer" containerID="6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba" Oct 30 07:42:47.516259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.516102 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" event={"ID":"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf","Type":"ContainerStarted","Data":"2adfbd23fc16548441491bb5e5953f99d351e01905ca7e4f27ec56e06bc60a81"} Oct 30 07:42:47.516259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.516177 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" event={"ID":"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf","Type":"ContainerStarted","Data":"98a77a5afc31ca53c11460553347734a83b07a232220eb13783faf55f052b775"} Oct 30 07:42:47.516259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.516215 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:47.531270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.531243 5894 scope.go:122] "RemoveContainer" containerID="6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba" Oct 30 07:42:47.531623 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:42:47.531599 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba\": container with ID starting with 6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba not found: ID does not exist" containerID="6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba" Oct 30 07:42:47.531714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.531646 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba"} err="failed to get container status \"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba\": rpc error: code = NotFound desc = could not find container \"6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba\": container with ID starting with 6f5435d16f2776b633d43b6d7b303e7d4a6bed64695d76ea05aa88f7225ce0ba not found: ID does not exist" Oct 30 07:42:47.571418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.571287 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:42:47.577410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.577377 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-847555f7c-47r6d"] Oct 30 07:42:47.615443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.615363 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" podStartSLOduration=1.61534424 podStartE2EDuration="1.61534424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:47.614017979 +0000 UTC m=+2731.304935416" watchObservedRunningTime="2026-10-30 07:42:47.61534424 +0000 UTC m=+2731.306261677" Oct 30 07:42:47.680661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.675214 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:42:47.685794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.682132 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:47.686590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.686565 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:47.692697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.690135 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:47.796112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:47.796073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:48.382443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:48.382403 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:48.433794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:48.433759 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" path="/var/lib/kubelet/pods/4563d70a-a3d5-41ee-b299-0f0fe1c19554/volumes" Oct 30 07:42:48.825190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:48.825155 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:49.363547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:49.363509 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:42:50.498227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:50.498173 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:50.537217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:50.537171 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:50.654592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:50.654559 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:51.637788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:51.637751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:51.760616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:51.760582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:52.680351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:52.680299 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" podUID="e07be748-1557-4949-a9c2-866cf00d94b9" containerName="payload-processing" containerID="cri-o://08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384" gracePeriod=25 Oct 30 07:42:52.687385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:52.687343 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" podUID="ba4a0931-f108-4971-8708-325cb2feacb9" containerName="payload-processing" containerID="cri-o://c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c" gracePeriod=25 Oct 30 07:42:52.703431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:52.703400 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:52.825130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:52.825097 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:53.056200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.056168 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:42:53.057420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.057391 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" containerName="payload-processing" Oct 30 07:42:53.057545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.057422 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" containerName="payload-processing" Oct 30 07:42:53.057733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.057714 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4563d70a-a3d5-41ee-b299-0f0fe1c19554" containerName="payload-processing" Oct 30 07:42:53.063135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.062987 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.087089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.087049 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:42:53.093975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.093936 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pb8d7\"" Oct 30 07:42:53.106032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.106002 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:42:53.125051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.125026 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:53.161875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.161839 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.162058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.162034 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.162216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.162194 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcmlg\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.162289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.162231 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.265749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265554 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\") pod \"e07be748-1557-4949-a9c2-866cf00d94b9\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " Oct 30 07:42:53.265749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265712 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\") pod \"e07be748-1557-4949-a9c2-866cf00d94b9\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " Oct 30 07:42:53.266007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265774 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\") pod \"ba4a0931-f108-4971-8708-325cb2feacb9\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " Oct 30 07:42:53.266007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265799 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\") pod \"ba4a0931-f108-4971-8708-325cb2feacb9\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " Oct 30 07:42:53.266007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265840 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\") pod \"e07be748-1557-4949-a9c2-866cf00d94b9\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " Oct 30 07:42:53.266007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265880 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\") pod \"ba4a0931-f108-4971-8708-325cb2feacb9\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " Oct 30 07:42:53.266007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.265972 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\") pod \"ba4a0931-f108-4971-8708-325cb2feacb9\" (UID: \"ba4a0931-f108-4971-8708-325cb2feacb9\") " Oct 30 07:42:53.266283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.266030 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\") pod \"e07be748-1557-4949-a9c2-866cf00d94b9\" (UID: \"e07be748-1557-4949-a9c2-866cf00d94b9\") " Oct 30 07:42:53.266343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.266286 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.291668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.266440 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.291668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.266567 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcmlg\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.291668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.266592 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.302602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.302405 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume" pod "e07be748-1557-4949-a9c2-866cf00d94b9" (UID: "e07be748-1557-4949-a9c2-866cf00d94b9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:53.302830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.302723 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume" pod "ba4a0931-f108-4971-8708-325cb2feacb9" (UID: "ba4a0931-f108-4971-8708-325cb2feacb9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:53.303747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.303720 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.305987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.303935 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.305987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.305617 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume" pod "ba4a0931-f108-4971-8708-325cb2feacb9" (UID: "ba4a0931-f108-4971-8708-325cb2feacb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:53.305987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.305792 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume" pod "e07be748-1557-4949-a9c2-866cf00d94b9" (UID: "e07be748-1557-4949-a9c2-866cf00d94b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:53.318193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.318151 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq" pod "ba4a0931-f108-4971-8708-325cb2feacb9" (UID: "ba4a0931-f108-4971-8708-325cb2feacb9"). InnerVolumeSpecName "kube-api-access-kndqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:53.333841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.333801 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials" pod "ba4a0931-f108-4971-8708-325cb2feacb9" (UID: "ba4a0931-f108-4971-8708-325cb2feacb9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:53.334077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.333856 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials" pod "e07be748-1557-4949-a9c2-866cf00d94b9" (UID: "e07be748-1557-4949-a9c2-866cf00d94b9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:53.343042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.342832 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.344235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.344170 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcmlg\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\") pod \"payload-processing-external-model-864476c44c-ckns5\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.345074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.345048 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962" pod "e07be748-1557-4949-a9c2-866cf00d94b9" (UID: "e07be748-1557-4949-a9c2-866cf00d94b9"). InnerVolumeSpecName "kube-api-access-fq962". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:53.368672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368622 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.368672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368663 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq962\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-kube-api-access-fq962\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.368672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368673 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.369016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368681 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.369016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368693 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07be748-1557-4949-a9c2-866cf00d94b9-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.369016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368702 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kndqq\" (UniqueName: \"kubernetes.io/projected/ba4a0931-f108-4971-8708-325cb2feacb9-kube-api-access-kndqq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.369016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368712 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba4a0931-f108-4971-8708-325cb2feacb9-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.369016 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.368726 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e07be748-1557-4949-a9c2-866cf00d94b9-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:53.423224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.423190 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:53.575626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.575592 5894 generic.go:356] "Generic (PLEG): container finished" podID="ba4a0931-f108-4971-8708-325cb2feacb9" containerID="c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c" exitCode=0 Oct 30 07:42:53.575626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.575649 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" Oct 30 07:42:53.575956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.575702 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" event={"ID":"ba4a0931-f108-4971-8708-325cb2feacb9","Type":"ContainerDied","Data":"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c"} Oct 30 07:42:53.575956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.575776 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr" event={"ID":"ba4a0931-f108-4971-8708-325cb2feacb9","Type":"ContainerDied","Data":"287befcd1542f604c0de8601a73726e4b96179d29c108c75f37ea6cf44dfedbd"} Oct 30 07:42:53.575956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.575811 5894 scope.go:122] "RemoveContainer" containerID="c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c" Oct 30 07:42:53.581143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.581113 5894 generic.go:356] "Generic (PLEG): container finished" podID="e07be748-1557-4949-a9c2-866cf00d94b9" containerID="08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384" exitCode=0 Oct 30 07:42:53.581307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.581284 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" event={"ID":"e07be748-1557-4949-a9c2-866cf00d94b9","Type":"ContainerDied","Data":"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384"} Oct 30 07:42:53.581372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.581343 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" event={"ID":"e07be748-1557-4949-a9c2-866cf00d94b9","Type":"ContainerDied","Data":"193b51583f29113c511046d3c4faa0c312ad81b7bddf7442bcb5d348654ce0b0"} Oct 30 07:42:53.581436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.581422 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj" Oct 30 07:42:53.591671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.591627 5894 scope.go:122] "RemoveContainer" containerID="c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c" Oct 30 07:42:53.592047 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:42:53.592024 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c\": container with ID starting with c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c not found: ID does not exist" containerID="c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c" Oct 30 07:42:53.592140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.592055 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c"} err="failed to get container status \"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c\": rpc error: code = NotFound desc = could not find container \"c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c\": container with ID starting with c58c0c6a74af68511c83c305f85e37f4b916d4940063dfb7472d7e55ae00d14c not found: ID does not exist" Oct 30 07:42:53.592140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.592071 5894 scope.go:122] "RemoveContainer" containerID="08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384" Oct 30 07:42:53.602980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.602954 5894 scope.go:122] "RemoveContainer" containerID="08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384" Oct 30 07:42:53.603345 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:42:53.603320 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384\": container with ID starting with 08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384 not found: ID does not exist" containerID="08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384" Oct 30 07:42:53.603433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.603354 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384"} err="failed to get container status \"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384\": rpc error: code = NotFound desc = could not find container \"08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384\": container with ID starting with 08841e4f849c6199067375543bf0e3ac1f1f3c22b297cc1735fbb9d9a4a3b384 not found: ID does not exist" Oct 30 07:42:53.632218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.632186 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:53.636163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.636138 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-9b5cclrhr"] Oct 30 07:42:53.671605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.671573 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:42:53.675428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.675400 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-5b98k5nbj"] Oct 30 07:42:53.730027 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:53.729996 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode325b5f4_cb50_44f3_839b_895c8a3db30a.slice/crio-143a1d6beecdb816b0683f95a3f353b42a45e0a7d9ddd504d92f75b73dbb37d6 WatchSource:0}: Error finding container 143a1d6beecdb816b0683f95a3f353b42a45e0a7d9ddd504d92f75b73dbb37d6: Status 404 returned error can't find the container with id 143a1d6beecdb816b0683f95a3f353b42a45e0a7d9ddd504d92f75b73dbb37d6 Oct 30 07:42:53.731669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:53.731623 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:42:54.280657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.280605 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv"] Oct 30 07:42:54.283301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.283270 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e07be748-1557-4949-a9c2-866cf00d94b9" containerName="payload-processing" Oct 30 07:42:54.283538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.283513 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07be748-1557-4949-a9c2-866cf00d94b9" containerName="payload-processing" Oct 30 07:42:54.283735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.283719 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ba4a0931-f108-4971-8708-325cb2feacb9" containerName="payload-processing" Oct 30 07:42:54.283808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.283736 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba4a0931-f108-4971-8708-325cb2feacb9" containerName="payload-processing" Oct 30 07:42:54.284063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.284047 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba4a0931-f108-4971-8708-325cb2feacb9" containerName="payload-processing" Oct 30 07:42:54.284127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.284069 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07be748-1557-4949-a9c2-866cf00d94b9" containerName="payload-processing" Oct 30 07:42:54.291455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.291424 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.304459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.304427 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv"] Oct 30 07:42:54.307568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.307534 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:42:54.307568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.307551 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:42:54.307781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.307532 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:42:54.307781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.307603 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:42:54.386671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.386595 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.386836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.386803 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.386931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.386913 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw59\" (UniqueName: \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.433600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.433542 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:54.434817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.434788 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba4a0931-f108-4971-8708-325cb2feacb9" path="/var/lib/kubelet/pods/ba4a0931-f108-4971-8708-325cb2feacb9/volumes" Oct 30 07:42:54.435379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.435358 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07be748-1557-4949-a9c2-866cf00d94b9" path="/var/lib/kubelet/pods/e07be748-1557-4949-a9c2-866cf00d94b9/volumes" Oct 30 07:42:54.502551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.502508 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.503676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.503617 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.506203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.506179 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw59\" (UniqueName: \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.507223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.504989 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.509350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.509325 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.547709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.544543 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw59\" (UniqueName: \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\") pod \"maas-api-revoke-sub-a64e105e-gcdjv\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.597388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.597346 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" event={"ID":"e325b5f4-cb50-44f3-839b-895c8a3db30a","Type":"ContainerStarted","Data":"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0"} Oct 30 07:42:54.597697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.597673 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" event={"ID":"e325b5f4-cb50-44f3-839b-895c8a3db30a","Type":"ContainerStarted","Data":"143a1d6beecdb816b0683f95a3f353b42a45e0a7d9ddd504d92f75b73dbb37d6"} Oct 30 07:42:54.610733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.610691 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:54.611618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.611589 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:54.657854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.657795 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" podStartSLOduration=1.657779855 podStartE2EDuration="1.657779855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:54.653717367 +0000 UTC m=+2738.344634803" watchObservedRunningTime="2026-10-30 07:42:54.657779855 +0000 UTC m=+2738.348697291" Oct 30 07:42:54.935629 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:42:54.935600 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce887b3_2ac0_4009_ae0e_805bd30dc1e4.slice/crio-01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c WatchSource:0}: Error finding container 01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c: Status 404 returned error can't find the container with id 01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c Oct 30 07:42:54.936967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:54.936941 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv"] Oct 30 07:42:55.020433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.020390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:55.142972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.142933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:55.268245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.268127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:55.500086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.500043 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" podUID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" containerName="payload-processing" containerID="cri-o://2adfbd23fc16548441491bb5e5953f99d351e01905ca7e4f27ec56e06bc60a81" gracePeriod=25 Oct 30 07:42:55.629999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.629963 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" event={"ID":"cce887b3-2ac0-4009-ae0e-805bd30dc1e4","Type":"ContainerStarted","Data":"e51dc912644d5089793c83603fda53319eaba192eac4c568c12294d869e2298a"} Oct 30 07:42:55.630169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.630021 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" event={"ID":"cce887b3-2ac0-4009-ae0e-805bd30dc1e4","Type":"ContainerStarted","Data":"01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c"} Oct 30 07:42:55.632244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.632216 5894 generic.go:356] "Generic (PLEG): container finished" podID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" containerID="2adfbd23fc16548441491bb5e5953f99d351e01905ca7e4f27ec56e06bc60a81" exitCode=0 Oct 30 07:42:55.632389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.632284 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" event={"ID":"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf","Type":"ContainerDied","Data":"2adfbd23fc16548441491bb5e5953f99d351e01905ca7e4f27ec56e06bc60a81"} Oct 30 07:42:55.681608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.679130 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" podStartSLOduration=1.679111962 podStartE2EDuration="1.679111962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:42:55.676772487 +0000 UTC m=+2739.367689955" watchObservedRunningTime="2026-10-30 07:42:55.679111962 +0000 UTC m=+2739.370029402" Oct 30 07:42:55.785713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.785611 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:55.831742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.831695 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\") pod \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " Oct 30 07:42:55.831956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.831759 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\") pod \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " Oct 30 07:42:55.831956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.831943 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\") pod \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " Oct 30 07:42:55.832173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.832066 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\") pod \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\" (UID: \"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf\") " Oct 30 07:42:55.833852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.833824 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume" pod "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" (UID: "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:55.834486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.834461 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume" pod "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" (UID: "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:55.834714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.834692 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458" pod "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" (UID: "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf"). InnerVolumeSpecName "kube-api-access-b5458". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:55.834809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.834724 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials" pod "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" (UID: "d06c5120-8e56-46a7-b3fb-0e1f8fab88bf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:55.934438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.934402 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:55.934438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.934433 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5458\" (UniqueName: \"kubernetes.io/projected/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-kube-api-access-b5458\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:55.934438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.934444 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:55.934721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:55.934458 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:56.255049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.254958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:56.641164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.641133 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" Oct 30 07:42:56.641357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.641167 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-xtvlx" event={"ID":"d06c5120-8e56-46a7-b3fb-0e1f8fab88bf","Type":"ContainerDied","Data":"98a77a5afc31ca53c11460553347734a83b07a232220eb13783faf55f052b775"} Oct 30 07:42:56.641357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.641230 5894 scope.go:122] "RemoveContainer" containerID="2adfbd23fc16548441491bb5e5953f99d351e01905ca7e4f27ec56e06bc60a81" Oct 30 07:42:56.643181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.643150 5894 generic.go:356] "Generic (PLEG): container finished" podID="cce887b3-2ac0-4009-ae0e-805bd30dc1e4" containerID="e51dc912644d5089793c83603fda53319eaba192eac4c568c12294d869e2298a" exitCode=0 Oct 30 07:42:56.643314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.643223 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" event={"ID":"cce887b3-2ac0-4009-ae0e-805bd30dc1e4","Type":"ContainerDied","Data":"e51dc912644d5089793c83603fda53319eaba192eac4c568c12294d869e2298a"} Oct 30 07:42:56.737011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.736977 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:56.741316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:56.741285 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-xtvlx"] Oct 30 07:42:57.819068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.819041 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:57.860425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.860398 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\") pod \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " Oct 30 07:42:57.860609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.860462 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\" (UniqueName: \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\") pod \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " Oct 30 07:42:57.860699 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.860666 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\") pod \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\" (UID: \"cce887b3-2ac0-4009-ae0e-805bd30dc1e4\") " Oct 30 07:42:57.860912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.860853 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca" pod "cce887b3-2ac0-4009-ae0e-805bd30dc1e4" (UID: "cce887b3-2ac0-4009-ae0e-805bd30dc1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:42:57.861292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.861257 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:57.863193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.863169 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca" pod "cce887b3-2ac0-4009-ae0e-805bd30dc1e4" (UID: "cce887b3-2ac0-4009-ae0e-805bd30dc1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:42:57.863354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.863288 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59" pod "cce887b3-2ac0-4009-ae0e-805bd30dc1e4" (UID: "cce887b3-2ac0-4009-ae0e-805bd30dc1e4"). InnerVolumeSpecName "kube-api-access-9bw59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:42:57.915068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.914913 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:57.965760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.965699 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:57.966050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:57.966024 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bw59\" (UniqueName: \"kubernetes.io/projected/cce887b3-2ac0-4009-ae0e-805bd30dc1e4-kube-api-access-9bw59\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:42:58.129503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.129464 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:42:58.136905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.136880 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:42:58.293105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.293072 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:58.427324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.425598 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:42:58.439713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.439330 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" path="/var/lib/kubelet/pods/d06c5120-8e56-46a7-b3fb-0e1f8fab88bf/volumes" Oct 30 07:42:58.681409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.681312 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" event={"ID":"cce887b3-2ac0-4009-ae0e-805bd30dc1e4","Type":"ContainerDied","Data":"01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c"} Oct 30 07:42:58.682806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.681726 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv" Oct 30 07:42:58.682985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:58.682796 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01efb7c642d360107fb381a29f24d49210c2a2b363e546e8a02d56726952aa1c" Oct 30 07:42:59.762508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:42:59.762466 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:00.003267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:00.003221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:00.233032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:00.233000 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:00.350824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:00.350788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:01.488372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.488335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:01.626658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.621597 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:01.746666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.746547 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v"] Oct 30 07:43:01.748351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748318 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" containerName="payload-processing" Oct 30 07:43:01.748351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748348 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" containerName="payload-processing" Oct 30 07:43:01.748563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748368 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cce887b3-2ac0-4009-ae0e-805bd30dc1e4" containerName="revoke-keys" Oct 30 07:43:01.748563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748377 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cce887b3-2ac0-4009-ae0e-805bd30dc1e4" containerName="revoke-keys" Oct 30 07:43:01.748825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748805 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d06c5120-8e56-46a7-b3fb-0e1f8fab88bf" containerName="payload-processing" Oct 30 07:43:01.748902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.748831 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cce887b3-2ac0-4009-ae0e-805bd30dc1e4" containerName="revoke-keys" Oct 30 07:43:01.754793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.754766 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.757406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.757379 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v2b4\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.757556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.757493 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.757627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.757612 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.757721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.757704 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.770001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.769971 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v"] Oct 30 07:43:01.779178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.779145 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-2v9m4\"" Oct 30 07:43:01.779341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.779233 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-bf4fecc91e60\"" Oct 30 07:43:01.866661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.866456 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.866661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.866651 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.866981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.866748 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.866981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.866923 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v2b4\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.888689 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:01.867887 5894 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4fecc91e60: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:01.888689 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:01.867915 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:01.888689 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:01.868004 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials podName:72f5b398-8b7a-4d2b-a532-20b2f6a13e03 nodeName:}" failed. No retries permitted until 2026-10-30 07:43:02.367981564 +0000 UTC m=+2746.058898992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials") pod "payload-processing-external-model-55dcb64cb5-szv8v" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03") : secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:01.888689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.871658 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.888689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.872326 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:01.919459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:01.907869 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v2b4\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:02.019606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.019498 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v"] Oct 30 07:43:02.024335 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:02.024294 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" podUID="72f5b398-8b7a-4d2b-a532-20b2f6a13e03" Oct 30 07:43:02.094814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.094770 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:02.388662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.378912 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:02.388662 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:02.379331 5894 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4fecc91e60: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:02.388662 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:02.379352 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:02.388662 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:02.379413 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials podName:72f5b398-8b7a-4d2b-a532-20b2f6a13e03 nodeName:}" failed. No retries permitted until 2026-10-30 07:43:03.379394242 +0000 UTC m=+2747.070311670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials") pod "payload-processing-external-model-55dcb64cb5-szv8v" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03") : secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:02.575894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.575861 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:02.711120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.711026 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:02.719487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.719455 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:02.729780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.729752 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:02.794726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.794689 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\") pod \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " Oct 30 07:43:02.794930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.794846 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\") pod \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " Oct 30 07:43:02.795011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.794966 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\") pod \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " Oct 30 07:43:02.795291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.795265 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume" pod "72f5b398-8b7a-4d2b-a532-20b2f6a13e03" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:02.795371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.795341 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume" pod "72f5b398-8b7a-4d2b-a532-20b2f6a13e03" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:02.795519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.795503 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:02.795564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.795522 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:02.797190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.797163 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4" pod "72f5b398-8b7a-4d2b-a532-20b2f6a13e03" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03"). InnerVolumeSpecName "kube-api-access-9v2b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:02.846149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.846113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:02.896778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:02.896708 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v2b4\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-kube-api-access-9v2b4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:03.044848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.044813 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:03.130544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.130495 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" podUID="e325b5f4-cb50-44f3-839b-895c8a3db30a" containerName="payload-processing" containerID="cri-o://8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0" gracePeriod=25 Oct 30 07:43:03.209203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.209164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:03.338711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.338266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:03.422657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.407734 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials\") pod \"payload-processing-external-model-55dcb64cb5-szv8v\" (UID: \"72f5b398-8b7a-4d2b-a532-20b2f6a13e03\") " pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:03.422657 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:03.407986 5894 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-bf4fecc91e60: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:03.422657 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:03.408007 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v: secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:03.422657 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:03.408068 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials podName:72f5b398-8b7a-4d2b-a532-20b2f6a13e03 nodeName:}" failed. No retries permitted until 2026-10-30 07:43:05.408049631 +0000 UTC m=+2749.098967060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials") pod "payload-processing-external-model-55dcb64cb5-szv8v" (UID: "72f5b398-8b7a-4d2b-a532-20b2f6a13e03") : secret "agc-cred-bf4fecc91e60" not found Oct 30 07:43:03.488817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.488375 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:03.545095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.545062 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:43:03.616445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.616360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:03.713759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.713719 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\") pod \"e325b5f4-cb50-44f3-839b-895c8a3db30a\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " Oct 30 07:43:03.713958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.713800 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\") pod \"e325b5f4-cb50-44f3-839b-895c8a3db30a\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " Oct 30 07:43:03.713958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.713943 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\") pod \"e325b5f4-cb50-44f3-839b-895c8a3db30a\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " Oct 30 07:43:03.714103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.714084 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\") pod \"e325b5f4-cb50-44f3-839b-895c8a3db30a\" (UID: \"e325b5f4-cb50-44f3-839b-895c8a3db30a\") " Oct 30 07:43:03.714294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.714272 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume" pod "e325b5f4-cb50-44f3-839b-895c8a3db30a" (UID: "e325b5f4-cb50-44f3-839b-895c8a3db30a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:03.714370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.714347 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume" pod "e325b5f4-cb50-44f3-839b-895c8a3db30a" (UID: "e325b5f4-cb50-44f3-839b-895c8a3db30a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:03.714696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.714679 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:03.714750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.714698 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e325b5f4-cb50-44f3-839b-895c8a3db30a-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:03.715992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.715971 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials" pod "e325b5f4-cb50-44f3-839b-895c8a3db30a" (UID: "e325b5f4-cb50-44f3-839b-895c8a3db30a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:03.716068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.716017 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg" pod "e325b5f4-cb50-44f3-839b-895c8a3db30a" (UID: "e325b5f4-cb50-44f3-839b-895c8a3db30a"). InnerVolumeSpecName "kube-api-access-dcmlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:03.729559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729526 5894 generic.go:356] "Generic (PLEG): container finished" podID="e325b5f4-cb50-44f3-839b-895c8a3db30a" containerID="8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0" exitCode=0 Oct 30 07:43:03.729708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729576 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" Oct 30 07:43:03.729708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729600 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" event={"ID":"e325b5f4-cb50-44f3-839b-895c8a3db30a","Type":"ContainerDied","Data":"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0"} Oct 30 07:43:03.729708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729667 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-864476c44c-ckns5" event={"ID":"e325b5f4-cb50-44f3-839b-895c8a3db30a","Type":"ContainerDied","Data":"143a1d6beecdb816b0683f95a3f353b42a45e0a7d9ddd504d92f75b73dbb37d6"} Oct 30 07:43:03.729708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729690 5894 scope.go:122] "RemoveContainer" containerID="8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0" Oct 30 07:43:03.729936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.729924 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v" Oct 30 07:43:03.748920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.748898 5894 scope.go:122] "RemoveContainer" containerID="8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0" Oct 30 07:43:03.749211 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:03.749191 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0\": container with ID starting with 8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0 not found: ID does not exist" containerID="8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0" Oct 30 07:43:03.749284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.749223 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0"} err="failed to get container status \"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0\": rpc error: code = NotFound desc = could not find container \"8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0\": container with ID starting with 8c2b1f23ebd5cfcee4b6d8e336c1a19ef75b806e6d37e784ffe0ce067501e0e0 not found: ID does not exist" Oct 30 07:43:03.815611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.815575 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:03.815611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.815608 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcmlg\" (UniqueName: \"kubernetes.io/projected/e325b5f4-cb50-44f3-839b-895c8a3db30a-kube-api-access-dcmlg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:03.833155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.833123 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v"] Oct 30 07:43:03.836945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.836918 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-55dcb64cb5-szv8v"] Oct 30 07:43:03.872004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.871926 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:43:03.875311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.875285 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-864476c44c-ckns5"] Oct 30 07:43:03.916431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:03.916397 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f5b398-8b7a-4d2b-a532-20b2f6a13e03-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:04.183285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.183201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:04.315556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.315513 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:04.436725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.434512 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f5b398-8b7a-4d2b-a532-20b2f6a13e03" path="/var/lib/kubelet/pods/72f5b398-8b7a-4d2b-a532-20b2f6a13e03/volumes" Oct 30 07:43:04.436725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.434798 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:04.436725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.435034 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e325b5f4-cb50-44f3-839b-895c8a3db30a" path="/var/lib/kubelet/pods/e325b5f4-cb50-44f3-839b-895c8a3db30a/volumes" Oct 30 07:43:04.805398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:04.805363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:04.832771 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:04.832731 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:43:05.030211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.030178 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:05.520879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.520842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:05.677078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.677046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:05.808196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.808163 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:05.975769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.975728 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:05.976884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.976862 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e325b5f4-cb50-44f3-839b-895c8a3db30a" containerName="payload-processing" Oct 30 07:43:05.976959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.976887 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e325b5f4-cb50-44f3-839b-895c8a3db30a" containerName="payload-processing" Oct 30 07:43:05.977079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.977067 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e325b5f4-cb50-44f3-839b-895c8a3db30a" containerName="payload-processing" Oct 30 07:43:05.982858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:05.982835 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.023666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.008648 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:06.023666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.012177 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-e2e--f9d83428-dockercfg-n5qsc\"" Oct 30 07:43:06.023666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.016266 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"kube-root-ca.crt\"" Oct 30 07:43:06.023666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.016371 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b096ec\"" Oct 30 07:43:06.023666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.016442 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"agc-cred-aff03d9276a9\"" Oct 30 07:43:06.042472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.042431 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.042708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.042508 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.042785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.042708 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.042862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.042844 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ft4\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.104713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.104602 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"openshift-service-ca.crt\"" Oct 30 07:43:06.104713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.104615 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"routing-overlay\"" Oct 30 07:43:06.143971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.143929 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.144192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.144050 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ft4\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.144192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.144171 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.144334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.144220 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.144922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.144894 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.145072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.145039 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.147325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.147299 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.167929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.167897 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ft4\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.343662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.339938 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:06.680677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.680194 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:06.799144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.799105 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:06.881464 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:06.881429 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989a9d8d_4653_44ac_bc0b_2ee6cb99838e.slice/crio-e747ab7cd11cb8079c6b9283bb7fea427571faa1507e07c4c06d9eece8559209 WatchSource:0}: Error finding container e747ab7cd11cb8079c6b9283bb7fea427571faa1507e07c4c06d9eece8559209: Status 404 returned error can't find the container with id e747ab7cd11cb8079c6b9283bb7fea427571faa1507e07c4c06d9eece8559209 Oct 30 07:43:06.883242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.883217 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:06.931828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.931792 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:43:06.937384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.937354 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:06.957345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.957308 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:43:06.973901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.973868 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"payload-processing-external-model-plugins-e2e-extmod-a-a91a83\"" Oct 30 07:43:06.974057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.973904 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"kube-root-ca.crt\"" Oct 30 07:43:06.974057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.973983 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"routing-overlay\"" Oct 30 07:43:06.974232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:06.974199 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"openshift-service-ca.crt\"" Oct 30 07:43:07.010397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.010360 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"agc-cred-1f934860c122\"" Oct 30 07:43:07.010901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.010882 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"payload-processing-external-model-e2e--a4269818-dockercfg-v9k22\"" Oct 30 07:43:07.089656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.080423 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99cr\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.089656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.080509 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.089656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.080540 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.089656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.080591 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.183272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.182978 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.183272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.183033 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.183272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.183101 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.183749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.183404 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99cr\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.185222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.185194 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.186324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.186295 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.194673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.191304 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.209078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.209044 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99cr\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.280565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.280522 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.599606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.599577 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:43:07.602212 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:07.602184 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193aef98_5299_4ada_90bc_e33852b5562c.slice/crio-5974fd6295d415707eddbd32f1c08fc0b82073cde33d136dd0288e4635653ecf WatchSource:0}: Error finding container 5974fd6295d415707eddbd32f1c08fc0b82073cde33d136dd0288e4635653ecf: Status 404 returned error can't find the container with id 5974fd6295d415707eddbd32f1c08fc0b82073cde33d136dd0288e4635653ecf Oct 30 07:43:07.718399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.718365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:07.805412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.805359 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" event={"ID":"193aef98-5299-4ada-90bc-e33852b5562c","Type":"ContainerStarted","Data":"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965"} Oct 30 07:43:07.805621 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.805438 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" event={"ID":"193aef98-5299-4ada-90bc-e33852b5562c","Type":"ContainerStarted","Data":"5974fd6295d415707eddbd32f1c08fc0b82073cde33d136dd0288e4635653ecf"} Oct 30 07:43:07.806106 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.806083 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:07.808896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.808859 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" event={"ID":"989a9d8d-4653-44ac-bc0b-2ee6cb99838e","Type":"ContainerStarted","Data":"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032"} Oct 30 07:43:07.809044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.808910 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" event={"ID":"989a9d8d-4653-44ac-bc0b-2ee6cb99838e","Type":"ContainerStarted","Data":"e747ab7cd11cb8079c6b9283bb7fea427571faa1507e07c4c06d9eece8559209"} Oct 30 07:43:07.809121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.809095 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:07.852037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.851933 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" podStartSLOduration=1.8519174980000002 podStartE2EDuration="1.851917498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:07.848140546 +0000 UTC m=+2751.539057983" watchObservedRunningTime="2026-10-30 07:43:07.851917498 +0000 UTC m=+2751.542834934" Oct 30 07:43:07.859058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:07.859031 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:08.106966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.106844 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" podStartSLOduration=3.106820522 podStartE2EDuration="3.106820522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:07.895974931 +0000 UTC m=+2751.586892368" watchObservedRunningTime="2026-10-30 07:43:08.106820522 +0000 UTC m=+2751.797737956" Oct 30 07:43:08.110230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.110187 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:43:08.120706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.117030 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.134052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.133903 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"agc-cred-2671a401ff75\"" Oct 30 07:43:08.134052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.133989 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"kube-root-ca.crt\"" Oct 30 07:43:08.134052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.134007 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"openshift-service-ca.crt\"" Oct 30 07:43:08.134052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.133912 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-e2e--40e0c8c8-dockercfg-p4t2j\"" Oct 30 07:43:08.134052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.134022 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"routing-overlay\"" Oct 30 07:43:08.134441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.134100 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"payload-processing-external-model-plugins-e2e-extmod-b-b3a7d7\"" Oct 30 07:43:08.140682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.140246 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:43:08.234609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.234570 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897kv\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.234826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.234792 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.234899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.234825 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.234899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.234852 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.336590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.336542 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897kv\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.336805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.336753 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.336805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.336795 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.336918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.336834 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.337972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.337945 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.339314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.339287 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.341034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.340963 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.375695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.375599 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-897kv\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.470568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.470514 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:08.801008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.800979 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:43:08.801497 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:08.801467 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d04a175_f5c0_4850_8265_bfc4acfafb5b.slice/crio-92f77ebcb8b3c08b3cf3e376acf8a244503763cad1590ab9acfa024dfcfb9179 WatchSource:0}: Error finding container 92f77ebcb8b3c08b3cf3e376acf8a244503763cad1590ab9acfa024dfcfb9179: Status 404 returned error can't find the container with id 92f77ebcb8b3c08b3cf3e376acf8a244503763cad1590ab9acfa024dfcfb9179 Oct 30 07:43:08.822230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.822187 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" event={"ID":"7d04a175-f5c0-4850-8265-bfc4acfafb5b","Type":"ContainerStarted","Data":"92f77ebcb8b3c08b3cf3e376acf8a244503763cad1590ab9acfa024dfcfb9179"} Oct 30 07:43:08.839568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:08.839527 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:08.949750 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:08.949304 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:08.949750 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:08.949332 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:08.949750 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:08.949394 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:09.449372478 +0000 UTC m=+2753.140289908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:09.060363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.060262 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:09.344786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.344713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:09.469194 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:09.469152 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:09.469194 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:09.469189 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:09.469401 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:09.469247 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:10.469231305 +0000 UTC m=+2754.160148736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:09.807616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.807582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:09.832024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.831989 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" event={"ID":"7d04a175-f5c0-4850-8265-bfc4acfafb5b","Type":"ContainerStarted","Data":"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3"} Oct 30 07:43:09.832207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.832057 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:09.874362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.871557 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:43:09.914227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.913446 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" podStartSLOduration=1.913424242 podStartE2EDuration="1.913424242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:09.88419826 +0000 UTC m=+2753.575115696" watchObservedRunningTime="2026-10-30 07:43:09.913424242 +0000 UTC m=+2753.604341683" Oct 30 07:43:09.916843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.916810 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:43:09.917033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.916901 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:09.931936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:09.931907 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"e2e-cfl-94959b-provider-cred\"" Oct 30 07:43:10.086754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.086652 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.086754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.086707 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.087034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.086861 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.087034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.086976 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6lc\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.130247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.130215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:10.188208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.188171 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.188208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.188215 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.188494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.188305 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.188494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.188382 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6lc\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.189075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.189045 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.189174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.189157 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.190880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.190856 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.214023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.213986 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6lc\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.238190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.237913 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.249808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.249587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:10.493522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.493484 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:10.494506 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:10.494484 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:10.494506 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:10.494507 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:10.494610 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:10.494559 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:12.494545818 +0000 UTC m=+2756.185463253 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:10.618151 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:10.618121 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e7c955_d473_4bbb_9100_4ce3604e10a9.slice/crio-90f2aac9b0f1ddc655d2508f8773183520bad8c6c780e1f2ad95fd724e7ac1fb WatchSource:0}: Error finding container 90f2aac9b0f1ddc655d2508f8773183520bad8c6c780e1f2ad95fd724e7ac1fb: Status 404 returned error can't find the container with id 90f2aac9b0f1ddc655d2508f8773183520bad8c6c780e1f2ad95fd724e7ac1fb Oct 30 07:43:10.619659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.619619 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:43:10.769671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.764258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:10.871719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.864727 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" event={"ID":"d1e7c955-d473-4bbb-9100-4ce3604e10a9","Type":"ContainerStarted","Data":"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431"} Oct 30 07:43:10.871719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.864792 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" event={"ID":"d1e7c955-d473-4bbb-9100-4ce3604e10a9","Type":"ContainerStarted","Data":"90f2aac9b0f1ddc655d2508f8773183520bad8c6c780e1f2ad95fd724e7ac1fb"} Oct 30 07:43:10.871719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.864843 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:10.910034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:10.909977 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" podStartSLOduration=1.9099616350000002 podStartE2EDuration="1.909961635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:10.907738877 +0000 UTC m=+2754.598656308" watchObservedRunningTime="2026-10-30 07:43:10.909961635 +0000 UTC m=+2754.600879122" Oct 30 07:43:11.602252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:11.602216 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:12.530623 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:12.530577 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:12.530623 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:12.530616 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:12.530880 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:12.530707 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:16.530691284 +0000 UTC m=+2760.221608720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:13.219768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:13.219732 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:14.830185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:14.830153 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:16.500096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:16.500059 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:16.585229 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:16.585196 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:16.585229 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:16.585227 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:16.585445 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:16.585312 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:24.58529251 +0000 UTC m=+2768.276209947 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:17.216869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:17.216832 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:17.405717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:17.405682 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:17.530057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:17.530016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:18.122684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:18.122618 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:18.825704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:18.825672 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:43:18.826140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:18.825748 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:19.391926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:19.391892 5894 scope.go:122] "RemoveContainer" containerID="0fd71da603130d2af3f2e3e57492d6e348a28b1a78d342eea8b221154300bc8b" Oct 30 07:43:19.833352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:19.833320 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:20.867029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:20.866999 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:43:21.484057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:21.484022 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:22.886294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:22.886255 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:43:22.964128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:22.964092 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:23.127055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:23.127021 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:23.690568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:23.690524 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:23.830515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:23.830481 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:23.972288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:23.972201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:24.129442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:24.129408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:24.422740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:24.422708 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:24.613233 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:24.613190 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:24.613233 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:24.613219 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:24.613470 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:24.613274 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials podName:989a9d8d-4653-44ac-bc0b-2ee6cb99838e nodeName:}" failed. No retries permitted until 2026-10-30 07:43:40.61325953 +0000 UTC m=+2784.304176970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:43:24.814593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:24.814559 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:25.086165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:25.086077 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:25.227686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:25.227623 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:26.854233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:26.854196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:27.487882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.487844 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:27.752870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.752791 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:43:27.759515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.759489 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.777651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.777608 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:43:27.786161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.786127 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"agc-cred-186ad5634d58\"" Oct 30 07:43:27.786358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.786339 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-e2e--90fb40dc-dockercfg-jkj7j\"" Oct 30 07:43:27.786791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.786766 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"payload-processing-external-model-plugins-e2e-extmod-a-705337\"" Oct 30 07:43:27.788658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.788619 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"routing-overlay\"" Oct 30 07:43:27.788658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.788650 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"kube-root-ca.crt\"" Oct 30 07:43:27.788802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.788715 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"openshift-service-ca.crt\"" Oct 30 07:43:27.890657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.883242 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.890657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.883330 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.890657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.883421 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.890657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.883473 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2gt7\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.966628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.966573 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" podUID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" containerName="payload-processing" containerID="cri-o://d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032" gracePeriod=25 Oct 30 07:43:27.988920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.988572 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.988920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.988682 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.992493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.990847 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.992493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.990922 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2gt7\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.992493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.992271 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.994276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.992318 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:27.999521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:27.999495 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:28.026730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.019689 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2gt7\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759cct2x\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:28.075083 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.075041 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:28.328193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.328167 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:28.391152 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:28.390939 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode914bca0_18ed_478b_a320_bfa8272bc806.slice/crio-4b8c786d40be6a0579069a332e64fa7c34317c587b432a181336117742c7dce1 WatchSource:0}: Error finding container 4b8c786d40be6a0579069a332e64fa7c34317c587b432a181336117742c7dce1: Status 404 returned error can't find the container with id 4b8c786d40be6a0579069a332e64fa7c34317c587b432a181336117742c7dce1 Oct 30 07:43:28.391152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.391059 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:43:28.419094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.419064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:28.560338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.558616 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\") pod \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " Oct 30 07:43:28.560338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.558790 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\") pod \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " Oct 30 07:43:28.560338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.558909 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\") pod \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " Oct 30 07:43:28.560338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.558990 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\") pod \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\" (UID: \"989a9d8d-4653-44ac-bc0b-2ee6cb99838e\") " Oct 30 07:43:28.560338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.560010 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume" pod "989a9d8d-4653-44ac-bc0b-2ee6cb99838e" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:28.560814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.560679 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume" pod "989a9d8d-4653-44ac-bc0b-2ee6cb99838e" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:28.579182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.572216 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials" pod "989a9d8d-4653-44ac-bc0b-2ee6cb99838e" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:28.579182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.576793 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4" pod "989a9d8d-4653-44ac-bc0b-2ee6cb99838e" (UID: "989a9d8d-4653-44ac-bc0b-2ee6cb99838e"). InnerVolumeSpecName "kube-api-access-k8ft4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:28.647141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.647103 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:28.660999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.660967 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:28.660999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.660996 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ft4\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-kube-api-access-k8ft4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:28.660999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.661009 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:28.661281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.661019 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/989a9d8d-4653-44ac-bc0b-2ee6cb99838e-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:28.672933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.672901 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:43:28.673676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.673658 5894 cpu_manager.go:414] "containerMap: removing container" podUID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" containerName="payload-processing" Oct 30 07:43:28.673676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.673676 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" containerName="payload-processing" Oct 30 07:43:28.673831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.673822 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" containerName="payload-processing" Oct 30 07:43:28.678097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.678066 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.717950 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"routing-overlay\"" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.718154 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"payload-processing-external-model-e2e--4ef33b46-dockercfg-qkvv8\"" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.718201 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"agc-cred-68034f7ddc34\"" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.718276 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"kube-root-ca.crt\"" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.718812 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"openshift-service-ca.crt\"" Oct 30 07:43:28.720663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.718831 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bffe43\"" Oct 30 07:43:28.722996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.722962 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:43:28.762857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.762818 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.763072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.762874 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.763072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.762991 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvjn8\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.763188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.763111 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.864503 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.864562 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.864684 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvjn8\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.864763 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.866197 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.867703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.867019 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.873692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.872599 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:28.903512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:28.903475 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvjn8\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:29.030568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.030527 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:29.042613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.042535 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" event={"ID":"e914bca0-18ed-478b-a320-bfa8272bc806","Type":"ContainerStarted","Data":"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c"} Oct 30 07:43:29.042917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.042616 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" event={"ID":"e914bca0-18ed-478b-a320-bfa8272bc806","Type":"ContainerStarted","Data":"4b8c786d40be6a0579069a332e64fa7c34317c587b432a181336117742c7dce1"} Oct 30 07:43:29.042917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.042864 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:29.044314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.044288 5894 generic.go:356] "Generic (PLEG): container finished" podID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" containerID="d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032" exitCode=0 Oct 30 07:43:29.044463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.044339 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" Oct 30 07:43:29.044463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.044353 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" event={"ID":"989a9d8d-4653-44ac-bc0b-2ee6cb99838e","Type":"ContainerDied","Data":"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032"} Oct 30 07:43:29.044463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.044385 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9" event={"ID":"989a9d8d-4653-44ac-bc0b-2ee6cb99838e","Type":"ContainerDied","Data":"e747ab7cd11cb8079c6b9283bb7fea427571faa1507e07c4c06d9eece8559209"} Oct 30 07:43:29.044463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.044405 5894 scope.go:122] "RemoveContainer" containerID="d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032" Oct 30 07:43:29.079775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.079744 5894 scope.go:122] "RemoveContainer" containerID="d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032" Oct 30 07:43:29.080370 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:29.080303 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032\": container with ID starting with d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032 not found: ID does not exist" containerID="d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032" Oct 30 07:43:29.080370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.080350 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032"} err="failed to get container status \"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032\": rpc error: code = NotFound desc = could not find container \"d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032\": container with ID starting with d1ea78fd295947f3ebcb74e1c0402b6a5e7123f86e00029721a4970ec952c032 not found: ID does not exist" Oct 30 07:43:29.087600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.087557 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" podStartSLOduration=2.087544108 podStartE2EDuration="2.087544108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:29.085458528 +0000 UTC m=+2772.776375965" watchObservedRunningTime="2026-10-30 07:43:29.087544108 +0000 UTC m=+2772.778461546" Oct 30 07:43:29.130448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.130364 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:29.136338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.136307 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d47sgj9"] Oct 30 07:43:29.349673 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:29.349599 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321c6cff_1a60_4d16_a79f_dc322478a6de.slice/crio-c62e05e95922e31fc934a7623498d47a218a9a55361f31febeae1061fd71f40e WatchSource:0}: Error finding container c62e05e95922e31fc934a7623498d47a218a9a55361f31febeae1061fd71f40e: Status 404 returned error can't find the container with id c62e05e95922e31fc934a7623498d47a218a9a55361f31febeae1061fd71f40e Oct 30 07:43:29.352659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:29.352611 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:43:30.083661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.081286 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" event={"ID":"321c6cff-1a60-4d16-a79f-dc322478a6de","Type":"ContainerStarted","Data":"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe"} Oct 30 07:43:30.083661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.081354 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" event={"ID":"321c6cff-1a60-4d16-a79f-dc322478a6de","Type":"ContainerStarted","Data":"c62e05e95922e31fc934a7623498d47a218a9a55361f31febeae1061fd71f40e"} Oct 30 07:43:30.083661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.082651 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:30.131669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.129088 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" podStartSLOduration=2.12906892 podStartE2EDuration="2.12906892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:30.128593921 +0000 UTC m=+2773.819511359" watchObservedRunningTime="2026-10-30 07:43:30.12906892 +0000 UTC m=+2773.819986360" Oct 30 07:43:30.440217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.440132 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989a9d8d-4653-44ac-bc0b-2ee6cb99838e" path="/var/lib/kubelet/pods/989a9d8d-4653-44ac-bc0b-2ee6cb99838e/volumes" Oct 30 07:43:30.531088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.531051 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:30.808898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:30.808860 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:31.977396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:31.977354 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj"] Oct 30 07:43:31.983190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:31.983163 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.000227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.000197 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj"] Oct 30 07:43:32.008123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.008092 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:43:32.008374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.008092 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:43:32.008374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.008225 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:43:32.011661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.010788 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:43:32.026564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.026532 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwc2x\" (UniqueName: \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.026564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.026574 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.026943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.026916 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.128190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.128157 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.128380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.128271 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwc2x\" (UniqueName: \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.128380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.128293 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.128894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.128865 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.130893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.130861 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.152431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.152404 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwc2x\" (UniqueName: \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\") pod \"maas-api-revoke-sub-01c68a06-l55lj\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.238920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.238794 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:32.308240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.303950 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:32.629340 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:32.629299 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6289d100_cdda_4627_83fc_5d98ba37e8bf.slice/crio-75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc WatchSource:0}: Error finding container 75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc: Status 404 returned error can't find the container with id 75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc Oct 30 07:43:32.631184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:32.631151 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj"] Oct 30 07:43:33.124463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:33.124426 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" event={"ID":"6289d100-cdda-4627-83fc-5d98ba37e8bf","Type":"ContainerStarted","Data":"0efd45ea88337281d92fa5f1edfdefbb1d210b64175f38e816f9470293c3faf0"} Oct 30 07:43:33.124904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:33.124482 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" event={"ID":"6289d100-cdda-4627-83fc-5d98ba37e8bf","Type":"ContainerStarted","Data":"75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc"} Oct 30 07:43:33.168701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:33.168571 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" podStartSLOduration=2.168551893 podStartE2EDuration="2.168551893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:33.166488558 +0000 UTC m=+2776.857405996" watchObservedRunningTime="2026-10-30 07:43:33.168551893 +0000 UTC m=+2776.859469331" Oct 30 07:43:33.954260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:33.954164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:34.133797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:34.133758 5894 generic.go:356] "Generic (PLEG): container finished" podID="6289d100-cdda-4627-83fc-5d98ba37e8bf" containerID="0efd45ea88337281d92fa5f1edfdefbb1d210b64175f38e816f9470293c3faf0" exitCode=0 Oct 30 07:43:34.134217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:34.133848 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" event={"ID":"6289d100-cdda-4627-83fc-5d98ba37e8bf","Type":"ContainerDied","Data":"0efd45ea88337281d92fa5f1edfdefbb1d210b64175f38e816f9470293c3faf0"} Oct 30 07:43:34.829561 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:43:34.829523 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:43:35.310405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.310372 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:35.375293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.375244 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\") pod \"6289d100-cdda-4627-83fc-5d98ba37e8bf\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " Oct 30 07:43:35.375489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.375323 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\") pod \"6289d100-cdda-4627-83fc-5d98ba37e8bf\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " Oct 30 07:43:35.375489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.375429 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\" (UniqueName: \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\") pod \"6289d100-cdda-4627-83fc-5d98ba37e8bf\" (UID: \"6289d100-cdda-4627-83fc-5d98ba37e8bf\") " Oct 30 07:43:35.376131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.376076 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca" pod "6289d100-cdda-4627-83fc-5d98ba37e8bf" (UID: "6289d100-cdda-4627-83fc-5d98ba37e8bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:35.385815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.385789 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca" pod "6289d100-cdda-4627-83fc-5d98ba37e8bf" (UID: "6289d100-cdda-4627-83fc-5d98ba37e8bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:43:35.385951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.385820 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x" pod "6289d100-cdda-4627-83fc-5d98ba37e8bf" (UID: "6289d100-cdda-4627-83fc-5d98ba37e8bf"). InnerVolumeSpecName "kube-api-access-kwc2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:35.478250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.478208 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:35.478250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.478240 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6289d100-cdda-4627-83fc-5d98ba37e8bf-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:35.478250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.478251 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwc2x\" (UniqueName: \"kubernetes.io/projected/6289d100-cdda-4627-83fc-5d98ba37e8bf-kube-api-access-kwc2x\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:35.764206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:35.764106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:36.150296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:36.150266 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" Oct 30 07:43:36.150296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:36.150277 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj" event={"ID":"6289d100-cdda-4627-83fc-5d98ba37e8bf","Type":"ContainerDied","Data":"75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc"} Oct 30 07:43:36.150527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:36.150346 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75ed8f494dcaf1e0ceeef278fb4df47a097c8b79b6efb264da2dbdac3ec5adcc" Oct 30 07:43:37.425682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:37.425626 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:39.099751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:39.099718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:40.098773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:40.098740 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:43:40.915936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:40.915900 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:42.118097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.118059 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:43:42.686647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.686589 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:42.782170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.782120 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5"] Oct 30 07:43:42.783552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.783520 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6289d100-cdda-4627-83fc-5d98ba37e8bf" containerName="revoke-keys" Oct 30 07:43:42.783741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.783725 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6289d100-cdda-4627-83fc-5d98ba37e8bf" containerName="revoke-keys" Oct 30 07:43:42.784133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.784116 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6289d100-cdda-4627-83fc-5d98ba37e8bf" containerName="revoke-keys" Oct 30 07:43:42.789187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.789161 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.804420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.804393 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5"] Oct 30 07:43:42.818747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.818715 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:43:42.829480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.829442 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:43:42.829480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.829456 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:43:42.829749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.829516 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:43:42.874795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.874762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:42.892549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.892517 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswdw\" (UniqueName: \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.892777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.892650 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.892828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.892785 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.992423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.992337 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:42.993768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.993741 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.993877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.993866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswdw\" (UniqueName: \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.993936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.993924 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.994372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.994348 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:42.996423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:42.996399 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:43.020660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:43.020595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswdw\" (UniqueName: \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\") pod \"maas-api-revoke-sub-1c34e81e-xdhq5\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:43.107668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:43.107622 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:43.412889 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:43.412856 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381c4e87_8621_415b_a5cc_1c072b5b90f9.slice/crio-a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109 WatchSource:0}: Error finding container a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109: Status 404 returned error can't find the container with id a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109 Oct 30 07:43:43.416097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:43.416066 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5"] Oct 30 07:43:44.227161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:44.227065 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" event={"ID":"381c4e87-8621-415b-a5cc-1c072b5b90f9","Type":"ContainerStarted","Data":"37b2e50fbe9238ed92bf0e44b9da410c024ccbce8d76c2aea923f26acf17eb8d"} Oct 30 07:43:44.227161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:44.227121 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" event={"ID":"381c4e87-8621-415b-a5cc-1c072b5b90f9","Type":"ContainerStarted","Data":"a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109"} Oct 30 07:43:44.314495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:44.314440 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" podStartSLOduration=2.314424242 podStartE2EDuration="2.314424242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:44.312019263 +0000 UTC m=+2788.002936701" watchObservedRunningTime="2026-10-30 07:43:44.314424242 +0000 UTC m=+2788.005341679" Oct 30 07:43:44.467985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:44.467950 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:45.236314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:45.236276 5894 generic.go:356] "Generic (PLEG): container finished" podID="381c4e87-8621-415b-a5cc-1c072b5b90f9" containerID="37b2e50fbe9238ed92bf0e44b9da410c024ccbce8d76c2aea923f26acf17eb8d" exitCode=0 Oct 30 07:43:45.236500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:45.236354 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" event={"ID":"381c4e87-8621-415b-a5cc-1c072b5b90f9","Type":"ContainerDied","Data":"37b2e50fbe9238ed92bf0e44b9da410c024ccbce8d76c2aea923f26acf17eb8d"} Oct 30 07:43:46.140134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.140096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:46.430678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.430608 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:46.443913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.443884 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\" (UniqueName: \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\") pod \"381c4e87-8621-415b-a5cc-1c072b5b90f9\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " Oct 30 07:43:46.444059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.444007 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\") pod \"381c4e87-8621-415b-a5cc-1c072b5b90f9\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " Oct 30 07:43:46.444131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.444119 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\") pod \"381c4e87-8621-415b-a5cc-1c072b5b90f9\" (UID: \"381c4e87-8621-415b-a5cc-1c072b5b90f9\") " Oct 30 07:43:46.444432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.444400 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca" pod "381c4e87-8621-415b-a5cc-1c072b5b90f9" (UID: "381c4e87-8621-415b-a5cc-1c072b5b90f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:43:46.446228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.446201 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw" pod "381c4e87-8621-415b-a5cc-1c072b5b90f9" (UID: "381c4e87-8621-415b-a5cc-1c072b5b90f9"). InnerVolumeSpecName "kube-api-access-wswdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:43:46.446337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.446207 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca" pod "381c4e87-8621-415b-a5cc-1c072b5b90f9" (UID: "381c4e87-8621-415b-a5cc-1c072b5b90f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:43:46.545424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.545379 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:46.545424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.545419 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wswdw\" (UniqueName: \"kubernetes.io/projected/381c4e87-8621-415b-a5cc-1c072b5b90f9-kube-api-access-wswdw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:46.545424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:46.545432 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381c4e87-8621-415b-a5cc-1c072b5b90f9-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:43:47.263438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:47.263406 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" Oct 30 07:43:47.263927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:47.263441 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5" event={"ID":"381c4e87-8621-415b-a5cc-1c072b5b90f9","Type":"ContainerDied","Data":"a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109"} Oct 30 07:43:47.263927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:47.263503 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a805b1a087f8cfc56ec2bbea1b60203503da1b952193adcbc4273f25ac840109" Oct 30 07:43:47.839866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:47.839836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:49.530141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:49.530105 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:51.210814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:51.210779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:52.981870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:52.981832 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:54.109787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.109742 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:54.242043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.242007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:54.488162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.488073 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:43:54.488915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.488896 5894 cpu_manager.go:414] "containerMap: removing container" podUID="381c4e87-8621-415b-a5cc-1c072b5b90f9" containerName="revoke-keys" Oct 30 07:43:54.488915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.488915 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381c4e87-8621-415b-a5cc-1c072b5b90f9" containerName="revoke-keys" Oct 30 07:43:54.489076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.489065 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="381c4e87-8621-415b-a5cc-1c072b5b90f9" containerName="revoke-keys" Oct 30 07:43:54.493202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.493182 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.512765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.512734 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:43:54.564724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.564681 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.565282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.564848 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.565282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.564906 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2f2\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.565282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.564950 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.666480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.666444 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.666730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.666527 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.666730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.666553 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2f2\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.666730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.666585 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.667320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.667291 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.667437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.667377 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.669176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.669144 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.691158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.691125 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2f2\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\") pod \"payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.807672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.807594 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:54.919126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:54.919090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:55.120367 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:43:55.118791 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d8a575_d655_4add_88c9_50e6e36540ca.slice/crio-69cc33cd4459ff7788cf9655ec18c6e1fd7bc69149badfdad2b3c711b6763dd1 WatchSource:0}: Error finding container 69cc33cd4459ff7788cf9655ec18c6e1fd7bc69149badfdad2b3c711b6763dd1: Status 404 returned error can't find the container with id 69cc33cd4459ff7788cf9655ec18c6e1fd7bc69149badfdad2b3c711b6763dd1 Oct 30 07:43:55.122561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.122530 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:43:55.187694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.187380 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:43:55.194276 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.194193 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:43:55.203502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.203473 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:55.318209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.318170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:55.334473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.334434 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" event={"ID":"a0d8a575-d655-4add-88c9-50e6e36540ca","Type":"ContainerStarted","Data":"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac"} Oct 30 07:43:55.334714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.334493 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" event={"ID":"a0d8a575-d655-4add-88c9-50e6e36540ca","Type":"ContainerStarted","Data":"69cc33cd4459ff7788cf9655ec18c6e1fd7bc69149badfdad2b3c711b6763dd1"} Oct 30 07:43:55.334714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.334521 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:43:55.380883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.380763 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" podStartSLOduration=1.380744218 podStartE2EDuration="1.380744218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:43:55.377262086 +0000 UTC m=+2799.068179524" watchObservedRunningTime="2026-10-30 07:43:55.380744218 +0000 UTC m=+2799.071661653" Oct 30 07:43:55.812234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:55.812204 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:56.725850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:56.725751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:57.344371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:57.344325 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:57.464909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:57.464875 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:43:58.372668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:43:58.362038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:00.038944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.038913 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:00.195396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.195350 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" podUID="274c887e-5217-455d-a409-2b4c7473a08c" containerName="payload-processing" containerID="cri-o://32b80457c905aeafdb12a58556b5bcd377ecc192b2b2520b24f7715338d06f48" gracePeriod=25 Oct 30 07:44:00.335593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.335549 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" podUID="a0d8a575-d655-4add-88c9-50e6e36540ca" containerName="payload-processing" containerID="cri-o://e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac" gracePeriod=25 Oct 30 07:44:00.404184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.404153 5894 generic.go:356] "Generic (PLEG): container finished" podID="274c887e-5217-455d-a409-2b4c7473a08c" containerID="32b80457c905aeafdb12a58556b5bcd377ecc192b2b2520b24f7715338d06f48" exitCode=0 Oct 30 07:44:00.404354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.404232 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" event={"ID":"274c887e-5217-455d-a409-2b4c7473a08c","Type":"ContainerDied","Data":"32b80457c905aeafdb12a58556b5bcd377ecc192b2b2520b24f7715338d06f48"} Oct 30 07:44:00.485442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.485415 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:44:00.567888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.567799 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\") pod \"274c887e-5217-455d-a409-2b4c7473a08c\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " Oct 30 07:44:00.567888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.567887 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\") pod \"274c887e-5217-455d-a409-2b4c7473a08c\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " Oct 30 07:44:00.568147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568070 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\") pod \"274c887e-5217-455d-a409-2b4c7473a08c\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " Oct 30 07:44:00.568217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568193 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume" pod "274c887e-5217-455d-a409-2b4c7473a08c" (UID: "274c887e-5217-455d-a409-2b4c7473a08c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:00.568285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568240 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\") pod \"274c887e-5217-455d-a409-2b4c7473a08c\" (UID: \"274c887e-5217-455d-a409-2b4c7473a08c\") " Oct 30 07:44:00.568560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568538 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume" pod "274c887e-5217-455d-a409-2b4c7473a08c" (UID: "274c887e-5217-455d-a409-2b4c7473a08c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:00.568840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568823 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.568918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.568845 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c887e-5217-455d-a409-2b4c7473a08c-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.573675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.571039 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv" pod "274c887e-5217-455d-a409-2b4c7473a08c" (UID: "274c887e-5217-455d-a409-2b4c7473a08c"). InnerVolumeSpecName "kube-api-access-p7blv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:00.573675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.571369 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials" pod "274c887e-5217-455d-a409-2b4c7473a08c" (UID: "274c887e-5217-455d-a409-2b4c7473a08c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:00.598042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.598007 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:44:00.670080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.670042 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\") pod \"a0d8a575-d655-4add-88c9-50e6e36540ca\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " Oct 30 07:44:00.670361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.670332 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\") pod \"a0d8a575-d655-4add-88c9-50e6e36540ca\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " Oct 30 07:44:00.670514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.670465 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\") pod \"a0d8a575-d655-4add-88c9-50e6e36540ca\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " Oct 30 07:44:00.670514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.670495 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\") pod \"a0d8a575-d655-4add-88c9-50e6e36540ca\" (UID: \"a0d8a575-d655-4add-88c9-50e6e36540ca\") " Oct 30 07:44:00.670702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.670584 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume" pod "a0d8a575-d655-4add-88c9-50e6e36540ca" (UID: "a0d8a575-d655-4add-88c9-50e6e36540ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:00.671058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.671031 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume" pod "a0d8a575-d655-4add-88c9-50e6e36540ca" (UID: "a0d8a575-d655-4add-88c9-50e6e36540ca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:00.671210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.671188 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7blv\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-kube-api-access-p7blv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.671294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.671216 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.671294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.671226 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a0d8a575-d655-4add-88c9-50e6e36540ca-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.671294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.671235 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/274c887e-5217-455d-a409-2b4c7473a08c-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.672907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.672877 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2" pod "a0d8a575-d655-4add-88c9-50e6e36540ca" (UID: "a0d8a575-d655-4add-88c9-50e6e36540ca"). InnerVolumeSpecName "kube-api-access-tk2f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:00.673006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.672985 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials" pod "a0d8a575-d655-4add-88c9-50e6e36540ca" (UID: "a0d8a575-d655-4add-88c9-50e6e36540ca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:00.772901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.772865 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2f2\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-kube-api-access-tk2f2\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:00.772901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:00.772896 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a0d8a575-d655-4add-88c9-50e6e36540ca-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:01.414305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.414265 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" event={"ID":"274c887e-5217-455d-a409-2b4c7473a08c","Type":"ContainerDied","Data":"7efcdf16e67a1d7c4366ee4dc86e862bb9245c9d4d6ab6fe72dc4f5ea8516f95"} Oct 30 07:44:01.414873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.414273 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg" Oct 30 07:44:01.414873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.414373 5894 scope.go:122] "RemoveContainer" containerID="32b80457c905aeafdb12a58556b5bcd377ecc192b2b2520b24f7715338d06f48" Oct 30 07:44:01.415887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.415862 5894 generic.go:356] "Generic (PLEG): container finished" podID="a0d8a575-d655-4add-88c9-50e6e36540ca" containerID="e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac" exitCode=0 Oct 30 07:44:01.415970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.415934 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" Oct 30 07:44:01.416030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.415934 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" event={"ID":"a0d8a575-d655-4add-88c9-50e6e36540ca","Type":"ContainerDied","Data":"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac"} Oct 30 07:44:01.416092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.416032 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr" event={"ID":"a0d8a575-d655-4add-88c9-50e6e36540ca","Type":"ContainerDied","Data":"69cc33cd4459ff7788cf9655ec18c6e1fd7bc69149badfdad2b3c711b6763dd1"} Oct 30 07:44:01.426920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.426900 5894 scope.go:122] "RemoveContainer" containerID="e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac" Oct 30 07:44:01.437115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.437091 5894 scope.go:122] "RemoveContainer" containerID="e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac" Oct 30 07:44:01.437414 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:01.437389 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac\": container with ID starting with e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac not found: ID does not exist" containerID="e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac" Oct 30 07:44:01.437492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.437430 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac"} err="failed to get container status \"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac\": rpc error: code = NotFound desc = could not find container \"e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac\": container with ID starting with e1b9df339e930afd6947549e2d7917fff599e80c837516ac5809233295617aac not found: ID does not exist" Oct 30 07:44:01.470079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.470046 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:44:01.482486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.482376 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-58b7vm5qr"] Oct 30 07:44:01.507489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.507454 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:44:01.514394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.514366 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1e0880/payload-processing-external-model-e2e-extmod-a-1e0880-6c7f7hldg"] Oct 30 07:44:01.736846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.736750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:01.815351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:01.815318 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:44:02.347671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.347626 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:44:02.348389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348374 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a0d8a575-d655-4add-88c9-50e6e36540ca" containerName="payload-processing" Oct 30 07:44:02.348432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348391 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0d8a575-d655-4add-88c9-50e6e36540ca" containerName="payload-processing" Oct 30 07:44:02.348432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348408 5894 cpu_manager.go:414] "containerMap: removing container" podUID="274c887e-5217-455d-a409-2b4c7473a08c" containerName="payload-processing" Oct 30 07:44:02.348432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348414 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274c887e-5217-455d-a409-2b4c7473a08c" containerName="payload-processing" Oct 30 07:44:02.348532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348508 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0d8a575-d655-4add-88c9-50e6e36540ca" containerName="payload-processing" Oct 30 07:44:02.348532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.348519 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="274c887e-5217-455d-a409-2b4c7473a08c" containerName="payload-processing" Oct 30 07:44:02.353919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.353896 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.387121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.387086 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:44:02.392968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.392940 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.429659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.428968 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:44:02.453603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.453572 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274c887e-5217-455d-a409-2b4c7473a08c" path="/var/lib/kubelet/pods/274c887e-5217-455d-a409-2b4c7473a08c/volumes" Oct 30 07:44:02.454172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.454149 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0d8a575-d655-4add-88c9-50e6e36540ca" path="/var/lib/kubelet/pods/a0d8a575-d655-4add-88c9-50e6e36540ca/volumes" Oct 30 07:44:02.454549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.454529 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:44:02.496092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496054 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496121 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496254 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496285 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496319 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4p4\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496404 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496431 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496455 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496569 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496603 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496668 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496745 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496779 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496800 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496814 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496876 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.496958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496919 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.496958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.496943 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.598656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598551 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.598830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598661 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.598830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598718 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.598830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598761 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.598830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598782 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.598830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598810 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4p4\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598862 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598887 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598918 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.598976 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599011 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599060 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599136 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599149 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599178 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599211 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599226 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599247 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.599517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599290 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599527 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599542 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599577 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599587 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599607 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599613 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599819 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599939 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.600063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.599978 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.602041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.601970 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.602474 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.602447 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.602573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.602542 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.602682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.602627 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.623164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.623134 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.623294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.623194 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.623398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.623373 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-vlkzh\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.623612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.623590 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4p4\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\") pod \"maas-default-gateway-openshift-default-6dcf757899-sgrmz\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:02.672183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.672139 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:02.761174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:02.757118 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:03.003763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.003704 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:03.020560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.020401 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:44:03.030210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.030174 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.030346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.030264 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.030346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.030306 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.167661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.159208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:03.311063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.310288 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.311063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.310392 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.311063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.310437 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:44:03.347672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.347061 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:44:03.551532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.550855 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" event={"ID":"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9","Type":"ContainerStarted","Data":"5791efabe78eb615a89c312f24e5ce2dc3fc1e84a548ef4a50b4e255caadb6cb"} Oct 30 07:44:03.566273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.565509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" event={"ID":"fd360e22-b736-427d-a64e-3e8c1c691cc7","Type":"ContainerStarted","Data":"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a"} Oct 30 07:44:03.566273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.565563 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" event={"ID":"fd360e22-b736-427d-a64e-3e8c1c691cc7","Type":"ContainerStarted","Data":"4dcb0dd370a2f4ab3b5f7c3f863444f702ba53e5eba432e6936575c16ba74208"} Oct 30 07:44:03.612721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.612620 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podStartSLOduration=1.612601575 podStartE2EDuration="1.612601575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:03.611994094 +0000 UTC m=+2807.302911536" watchObservedRunningTime="2026-10-30 07:44:03.612601575 +0000 UTC m=+2807.303519013" Oct 30 07:44:03.672866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.672829 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:03.803420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:03.803381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:04.624015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.623985 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" event={"ID":"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9","Type":"ContainerStarted","Data":"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23"} Oct 30 07:44:04.711657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.705831 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:04.711657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.705917 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:04.757895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.757860 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:04.759340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.759308 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:04.759493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:04.759358 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:04.854443 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:04.854405 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:44:05.642091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.642062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:05.676661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.675054 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:05.676661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.675130 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:05.757977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.757938 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:05.758190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.757992 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:05.809785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.809725 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podStartSLOduration=3.809704237 podStartE2EDuration="3.809704237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:04.702044693 +0000 UTC m=+2808.392962131" watchObservedRunningTime="2026-10-30 07:44:05.809704237 +0000 UTC m=+2809.500621675" Oct 30 07:44:05.817444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.817411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:05.821600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.821569 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp"] Oct 30 07:44:05.829679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.829063 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.849882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.849849 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6pm\" (UniqueName: \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.850264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.850235 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.850442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.850425 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.851715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.851358 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:44:05.852584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.852395 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:44:05.852584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.852459 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:44:05.852584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.852516 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:44:05.894983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.894847 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp"] Oct 30 07:44:05.955658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.954794 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6pm\" (UniqueName: \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.955658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.954907 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.955658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.954962 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:05.956013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:05.955902 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:06.043481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.041523 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:06.061439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.061379 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6pm\" (UniqueName: \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\") pod \"maas-api-revoke-sub-d8b664ca-wsvjp\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:06.165697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.165589 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:06.568731 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:06.568689 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaddd582_2419_4fe6_a557_2d5ce033a0d6.slice/crio-2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d WatchSource:0}: Error finding container 2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d: Status 404 returned error can't find the container with id 2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d Oct 30 07:44:06.579659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.570930 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp"] Oct 30 07:44:06.645707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.645676 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" event={"ID":"caddd582-2419-4fe6-a557-2d5ce033a0d6","Type":"ContainerStarted","Data":"2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d"} Oct 30 07:44:06.683661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.679065 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:06.683661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.679125 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:06.758687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.758624 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:06.758906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:06.758710 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:07.655849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.655811 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" event={"ID":"caddd582-2419-4fe6-a557-2d5ce033a0d6","Type":"ContainerStarted","Data":"dff15ee0a57c52437187d19d81654f982e8a8c304d3522879c232d0bad9d2a0b"} Oct 30 07:44:07.711657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.710957 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:07.711657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.711018 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:07.731660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.731229 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" podStartSLOduration=2.7312106529999998 podStartE2EDuration="2.731210653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:07.728482054 +0000 UTC m=+2811.419399493" watchObservedRunningTime="2026-10-30 07:44:07.731210653 +0000 UTC m=+2811.422128090" Oct 30 07:44:07.758021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.757968 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:07.758189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.758073 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:07.834169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:07.834133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:08.664913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.664875 5894 generic.go:356] "Generic (PLEG): container finished" podID="caddd582-2419-4fe6-a557-2d5ce033a0d6" containerID="dff15ee0a57c52437187d19d81654f982e8a8c304d3522879c232d0bad9d2a0b" exitCode=0 Oct 30 07:44:08.664913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.664917 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" event={"ID":"caddd582-2419-4fe6-a557-2d5ce033a0d6","Type":"ContainerDied","Data":"dff15ee0a57c52437187d19d81654f982e8a8c304d3522879c232d0bad9d2a0b"} Oct 30 07:44:08.719660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.713463 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:08.719660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.713523 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:08.758260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.758216 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:08.758461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:08.758271 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:09.417659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.417287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:09.539790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.539750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:09.692721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.676115 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:09.692721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.676178 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:09.723010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.722915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:09.762760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.762268 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:09.762760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.762324 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:09.909291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.909042 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:09.980130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:09.980098 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:10.006329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.006283 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\" (UniqueName: \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\") pod \"caddd582-2419-4fe6-a557-2d5ce033a0d6\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " Oct 30 07:44:10.006534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.006396 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\") pod \"caddd582-2419-4fe6-a557-2d5ce033a0d6\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " Oct 30 07:44:10.006534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.006475 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\") pod \"caddd582-2419-4fe6-a557-2d5ce033a0d6\" (UID: \"caddd582-2419-4fe6-a557-2d5ce033a0d6\") " Oct 30 07:44:10.007006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.006979 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca" pod "caddd582-2419-4fe6-a557-2d5ce033a0d6" (UID: "caddd582-2419-4fe6-a557-2d5ce033a0d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:10.007348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.007325 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:10.008813 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.008788 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm" pod "caddd582-2419-4fe6-a557-2d5ce033a0d6" (UID: "caddd582-2419-4fe6-a557-2d5ce033a0d6"). InnerVolumeSpecName "kube-api-access-pd6pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:10.009009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.008991 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca" pod "caddd582-2419-4fe6-a557-2d5ce033a0d6" (UID: "caddd582-2419-4fe6-a557-2d5ce033a0d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:44:10.108891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.108855 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd6pm\" (UniqueName: \"kubernetes.io/projected/caddd582-2419-4fe6-a557-2d5ce033a0d6-kube-api-access-pd6pm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:10.108891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.108891 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caddd582-2419-4fe6-a557-2d5ce033a0d6-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:10.597608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.597008 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:10.673654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.672854 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:10.673654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.672931 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:10.697558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.697531 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" Oct 30 07:44:10.698121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.697586 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp" event={"ID":"caddd582-2419-4fe6-a557-2d5ce033a0d6","Type":"ContainerDied","Data":"2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d"} Oct 30 07:44:10.698121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.697662 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2223c7be9e2f60b633ea0b571c7881e345978bd6d67ad0d6326bd34ed2bd939d" Oct 30 07:44:10.758592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.758560 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:10.758823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:10.758618 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:11.033041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.032996 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:11.672674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.672609 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-vlkzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body= Oct 30 07:44:11.672899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.672702 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" Oct 30 07:44:11.749592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.749535 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:11.758097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.758060 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-sgrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" start-of-body= Oct 30 07:44:11.758376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:11.758339 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.18:15021/healthz/ready\": dial tcp 10.134.1.18:15021: connect: connection refused" Oct 30 07:44:12.830978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.830942 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:12.835265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.835220 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:12.836761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.836736 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:44:12.932673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.932607 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:12.934010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.933983 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:12.938484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:12.938453 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:44:13.703896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:13.703859 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:14.342367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:14.342328 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:14.458642 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:14.458577 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:15.471207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:15.471165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:16.101159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.101126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:16.243003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.242944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:16.285018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.284984 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:44:16.285775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.285760 5894 cpu_manager.go:414] "containerMap: removing container" podUID="caddd582-2419-4fe6-a557-2d5ce033a0d6" containerName="revoke-keys" Oct 30 07:44:16.285837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.285776 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caddd582-2419-4fe6-a557-2d5ce033a0d6" containerName="revoke-keys" Oct 30 07:44:16.285897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.285887 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="caddd582-2419-4fe6-a557-2d5ce033a0d6" containerName="revoke-keys" Oct 30 07:44:16.289854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.289832 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.317561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.317525 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d5f604\"" Oct 30 07:44:16.317819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.317790 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d5f604-dockercfg-pzt7b\"" Oct 30 07:44:16.385864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.385777 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:44:16.394080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.394025 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.394418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.394381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnsf\" (UniqueName: \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.417767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.417732 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:44:16.501691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.497902 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.501691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.497981 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnsf\" (UniqueName: \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.534610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.534567 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d5f604\"" Oct 30 07:44:16.535577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.535533 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.559586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.558401 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnsf\" (UniqueName: \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\") pod \"payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.564872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.563323 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.588681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.584606 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:44:16.599669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.599604 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw4nq\" (UniqueName: \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.599851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.599828 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.618823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.618793 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d5f604-dockercfg-pzt7b\"" Oct 30 07:44:16.625040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.625012 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:16.648288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.648256 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc"] Oct 30 07:44:16.653534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.653490 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.669668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.669617 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:44:16.669849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.669691 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:44:16.669849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.669769 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:44:16.669849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.669833 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:44:16.673658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.673611 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc"] Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.701967 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.702019 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkl9\" (UniqueName: \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.702243 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.702287 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.702363 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw4nq\" (UniqueName: \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.704942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.703666 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.727916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.727833 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw4nq\" (UniqueName: \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\") pod \"payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.803650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.803605 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.803842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.803762 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.803936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.803900 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkl9\" (UniqueName: \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.804417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.804391 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.806313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.806283 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.827958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.827925 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkl9\" (UniqueName: \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\") pod \"maas-api-revoke-sub-85cbd019-vntqc\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:16.915153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.915054 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:16.944357 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:16.944324 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e0bbd8_5c3f_48ad_abdb_7fbfe47cb900.slice/crio-2be6a51913db02d77ac457db1e21ca92d6876c10c05e26d95f9781897ba7b6f9 WatchSource:0}: Error finding container 2be6a51913db02d77ac457db1e21ca92d6876c10c05e26d95f9781897ba7b6f9: Status 404 returned error can't find the container with id 2be6a51913db02d77ac457db1e21ca92d6876c10c05e26d95f9781897ba7b6f9 Oct 30 07:44:16.946104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.945982 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:44:16.999918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:16.999865 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:17.228439 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:17.228410 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15ae5eb_3dd9_4546_9e70_01fb89c57e51.slice/crio-348970bc9baea84d7a1b1ed11d5bbf8f85312e8395f2aa499e49c75b8c49723c WatchSource:0}: Error finding container 348970bc9baea84d7a1b1ed11d5bbf8f85312e8395f2aa499e49c75b8c49723c: Status 404 returned error can't find the container with id 348970bc9baea84d7a1b1ed11d5bbf8f85312e8395f2aa499e49c75b8c49723c Oct 30 07:44:17.229965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.229931 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:44:17.328363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.328278 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc"] Oct 30 07:44:17.425338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.425297 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:17.788643 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.788597 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" event={"ID":"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077","Type":"ContainerStarted","Data":"ab3d272233c8d73e981b270e1fbcb12639ab68255af2e08308286a7e2ccda3c2"} Oct 30 07:44:17.789203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.788712 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" event={"ID":"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077","Type":"ContainerStarted","Data":"1f666f1a0e3b0e70eed1962221f68928a85b9b91c3270a79806a857824d913d3"} Oct 30 07:44:17.790889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.790862 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" event={"ID":"e15ae5eb-3dd9-4546-9e70-01fb89c57e51","Type":"ContainerStarted","Data":"292ad0b6d2c7d7e8904470c93fd81f3bc631b1ae77ef70e268a7ef1fbccfc474"} Oct 30 07:44:17.791025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.790913 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" event={"ID":"e15ae5eb-3dd9-4546-9e70-01fb89c57e51","Type":"ContainerStarted","Data":"348970bc9baea84d7a1b1ed11d5bbf8f85312e8395f2aa499e49c75b8c49723c"} Oct 30 07:44:17.791025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.790946 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:17.792585 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.792557 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" event={"ID":"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900","Type":"ContainerStarted","Data":"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709"} Oct 30 07:44:17.792702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.792604 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" event={"ID":"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900","Type":"ContainerStarted","Data":"2be6a51913db02d77ac457db1e21ca92d6876c10c05e26d95f9781897ba7b6f9"} Oct 30 07:44:17.792702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.792667 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:17.831846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.831794 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" podStartSLOduration=1.8317806829999999 podStartE2EDuration="1.831780683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:17.830296619 +0000 UTC m=+2821.521214055" watchObservedRunningTime="2026-10-30 07:44:17.831780683 +0000 UTC m=+2821.522698120" Oct 30 07:44:17.875724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.875667 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" podStartSLOduration=1.8756525320000001 podStartE2EDuration="1.875652532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:17.871586241 +0000 UTC m=+2821.562503698" watchObservedRunningTime="2026-10-30 07:44:17.875652532 +0000 UTC m=+2821.566569966" Oct 30 07:44:17.922011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:17.921948 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" podStartSLOduration=1.921930947 podStartE2EDuration="1.921930947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:17.91757323 +0000 UTC m=+2821.608490667" watchObservedRunningTime="2026-10-30 07:44:17.921930947 +0000 UTC m=+2821.612848401" Oct 30 07:44:18.802159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:18.802115 5894 generic.go:356] "Generic (PLEG): container finished" podID="af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" containerID="ab3d272233c8d73e981b270e1fbcb12639ab68255af2e08308286a7e2ccda3c2" exitCode=0 Oct 30 07:44:18.802656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:18.802159 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" event={"ID":"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077","Type":"ContainerDied","Data":"ab3d272233c8d73e981b270e1fbcb12639ab68255af2e08308286a7e2ccda3c2"} Oct 30 07:44:19.134762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:19.134663 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:19.999266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:19.999236 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:20.056561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.056528 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\") pod \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " Oct 30 07:44:20.056561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.056563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\") pod \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " Oct 30 07:44:20.056956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.056932 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\" (UniqueName: \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\") pod \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\" (UID: \"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077\") " Oct 30 07:44:20.056956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.056945 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca" pod "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" (UID: "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:20.057605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.057580 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:20.059038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.058992 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca" pod "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" (UID: "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:44:20.059544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.059515 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9" pod "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" (UID: "af4b0c1f-4e27-47bf-83ea-2bc91bdb9077"). InnerVolumeSpecName "kube-api-access-qfkl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:20.158563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.158531 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkl9\" (UniqueName: \"kubernetes.io/projected/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-kube-api-access-qfkl9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:20.158563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.158561 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:20.811156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.811117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:20.818356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.818329 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" Oct 30 07:44:20.818514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.818358 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc" event={"ID":"af4b0c1f-4e27-47bf-83ea-2bc91bdb9077","Type":"ContainerDied","Data":"1f666f1a0e3b0e70eed1962221f68928a85b9b91c3270a79806a857824d913d3"} Oct 30 07:44:20.818514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:20.818405 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f666f1a0e3b0e70eed1962221f68928a85b9b91c3270a79806a857824d913d3" Oct 30 07:44:22.536095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:22.536054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:24.216953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:24.216909 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:25.910445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:25.910408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:27.743193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:27.743157 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:28.709353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.709315 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:28.832926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.832888 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:44:28.840727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.834718 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:44:28.840727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.837116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:28.926220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.926144 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2"] Oct 30 07:44:28.928018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.927887 5894 cpu_manager.go:414] "containerMap: removing container" podUID="af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" containerName="revoke-keys" Oct 30 07:44:28.928213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.928193 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" containerName="revoke-keys" Oct 30 07:44:28.928617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.928563 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" containerName="revoke-keys" Oct 30 07:44:28.935319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.935290 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:28.955957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.955921 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:44:28.956189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.956000 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:44:28.956189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.956044 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:44:28.956319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.956099 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:44:28.972321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:28.972275 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2"] Oct 30 07:44:29.037156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.037120 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lb2n\" (UniqueName: \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.037362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.037341 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.037509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.037492 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.139208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.139173 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.139421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.139281 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.139421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.139378 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lb2n\" (UniqueName: \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.140007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.139979 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.141878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.141854 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.158858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.158826 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:29.164057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.164032 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.172147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.172119 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lb2n\" (UniqueName: \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\") pod \"maas-api-revoke-sub-b6052151-q8ng2\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.183753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.183720 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:29.240902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.240819 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.241067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.240967 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22sp5\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.241067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.240994 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.241067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.241012 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.282134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.282094 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:29.348228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.348046 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.348331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.348248 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22sp5\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.348331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.348320 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.348451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.348371 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.349211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.349180 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.349951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.349920 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.353131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.353108 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.384701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.383956 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22sp5\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\") pod \"payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.500581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.500500 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:29.597986 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:29.597953 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e60fc68_daa8_4499_8586_5ca3c27e3771.slice/crio-8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b WatchSource:0}: Error finding container 8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b: Status 404 returned error can't find the container with id 8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b Oct 30 07:44:29.599883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.599854 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2"] Oct 30 07:44:29.713337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.713271 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:29.817408 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:29.817373 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c936f7_6dff_46e9_8b82_4415054e8aa4.slice/crio-a7fe89702331e9c4ac114aee0ac1da38124bdc88850787d8eb4a7c4ab0b50f1e WatchSource:0}: Error finding container a7fe89702331e9c4ac114aee0ac1da38124bdc88850787d8eb4a7c4ab0b50f1e: Status 404 returned error can't find the container with id a7fe89702331e9c4ac114aee0ac1da38124bdc88850787d8eb4a7c4ab0b50f1e Oct 30 07:44:29.827021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.825625 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:29.854618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.854574 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:44:29.859901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.859875 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:29.871741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.868853 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:29.919893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.919813 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" event={"ID":"7e60fc68-daa8-4499-8586-5ca3c27e3771","Type":"ContainerStarted","Data":"8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b"} Oct 30 07:44:29.921536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.921508 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" event={"ID":"a4c936f7-6dff-46e9-8b82-4415054e8aa4","Type":"ContainerStarted","Data":"a7fe89702331e9c4ac114aee0ac1da38124bdc88850787d8eb4a7c4ab0b50f1e"} Oct 30 07:44:29.988918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:29.988876 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:30.033488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.033397 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v"] Oct 30 07:44:30.037487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.037450 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab41498c-4mv5v"] Oct 30 07:44:30.041100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.041068 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g"] Oct 30 07:44:30.044430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.044404 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-62fc22bb-wqm2g"] Oct 30 07:44:30.436911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.436821 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42957b4b-773d-4ebc-a05b-774678b27ab7" path="/var/lib/kubelet/pods/42957b4b-773d-4ebc-a05b-774678b27ab7/volumes" Oct 30 07:44:30.437460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.437440 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c057c43-7296-4802-9b64-d346fe396036" path="/var/lib/kubelet/pods/8c057c43-7296-4802-9b64-d346fe396036/volumes" Oct 30 07:44:30.530385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.530346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:30.932327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.932294 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" event={"ID":"a4c936f7-6dff-46e9-8b82-4415054e8aa4","Type":"ContainerStarted","Data":"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e"} Oct 30 07:44:30.932831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.932561 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:30.934761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.934734 5894 generic.go:356] "Generic (PLEG): container finished" podID="7e60fc68-daa8-4499-8586-5ca3c27e3771" containerID="acf91373d2d52a9e48b7509f5361b2aebe5fdc819479fb2d45e11e5069fbe2b9" exitCode=0 Oct 30 07:44:30.934908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.934817 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" event={"ID":"7e60fc68-daa8-4499-8586-5ca3c27e3771","Type":"ContainerDied","Data":"acf91373d2d52a9e48b7509f5361b2aebe5fdc819479fb2d45e11e5069fbe2b9"} Oct 30 07:44:30.938947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.938923 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:30.981895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:30.981834 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" podStartSLOduration=1.981820913 podStartE2EDuration="1.981820913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:30.978058754 +0000 UTC m=+2834.668976190" watchObservedRunningTime="2026-10-30 07:44:30.981820913 +0000 UTC m=+2834.672738408" Oct 30 07:44:31.495286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:31.495251 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:32.132984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.132953 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:32.221162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.221067 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\") pod \"7e60fc68-daa8-4499-8586-5ca3c27e3771\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " Oct 30 07:44:32.221162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.221125 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\" (UniqueName: \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\") pod \"7e60fc68-daa8-4499-8586-5ca3c27e3771\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " Oct 30 07:44:32.221944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.221269 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\") pod \"7e60fc68-daa8-4499-8586-5ca3c27e3771\" (UID: \"7e60fc68-daa8-4499-8586-5ca3c27e3771\") " Oct 30 07:44:32.221944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.221518 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca" pod "7e60fc68-daa8-4499-8586-5ca3c27e3771" (UID: "7e60fc68-daa8-4499-8586-5ca3c27e3771"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:32.222072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.222002 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:32.223928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.223900 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n" pod "7e60fc68-daa8-4499-8586-5ca3c27e3771" (UID: "7e60fc68-daa8-4499-8586-5ca3c27e3771"). InnerVolumeSpecName "kube-api-access-7lb2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:32.224185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.224167 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca" pod "7e60fc68-daa8-4499-8586-5ca3c27e3771" (UID: "7e60fc68-daa8-4499-8586-5ca3c27e3771"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:44:32.323498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.323459 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e60fc68-daa8-4499-8586-5ca3c27e3771-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:32.323498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.323493 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lb2n\" (UniqueName: \"kubernetes.io/projected/7e60fc68-daa8-4499-8586-5ca3c27e3771-kube-api-access-7lb2n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:32.951720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.951691 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" Oct 30 07:44:32.951908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.951688 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2" event={"ID":"7e60fc68-daa8-4499-8586-5ca3c27e3771","Type":"ContainerDied","Data":"8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b"} Oct 30 07:44:32.951908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:32.951823 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f54bf2fe93148cfba781488e80fdeb5586210b00609f0cd3b12f931944a215b" Oct 30 07:44:33.232929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:33.232797 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:34.829653 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:34.829602 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:44:34.855554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:34.855515 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" podUID="69f122cc-8751-4b60-be36-ce4514ad9051" containerName="payload-processing" containerID="cri-o://5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0" gracePeriod=25 Oct 30 07:44:34.894594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:34.894567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:35.161190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.161162 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:44:35.266113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266076 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\") pod \"69f122cc-8751-4b60-be36-ce4514ad9051\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " Oct 30 07:44:35.266325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266180 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\") pod \"69f122cc-8751-4b60-be36-ce4514ad9051\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " Oct 30 07:44:35.266325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266206 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\") pod \"69f122cc-8751-4b60-be36-ce4514ad9051\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " Oct 30 07:44:35.266325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266226 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\") pod \"69f122cc-8751-4b60-be36-ce4514ad9051\" (UID: \"69f122cc-8751-4b60-be36-ce4514ad9051\") " Oct 30 07:44:35.266687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266585 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume" pod "69f122cc-8751-4b60-be36-ce4514ad9051" (UID: "69f122cc-8751-4b60-be36-ce4514ad9051"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:35.266811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266779 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume" pod "69f122cc-8751-4b60-be36-ce4514ad9051" (UID: "69f122cc-8751-4b60-be36-ce4514ad9051"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:35.266959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266941 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:35.266959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.266958 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f122cc-8751-4b60-be36-ce4514ad9051-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:35.268481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.268452 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97" pod "69f122cc-8751-4b60-be36-ce4514ad9051" (UID: "69f122cc-8751-4b60-be36-ce4514ad9051"). InnerVolumeSpecName "kube-api-access-bhq97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:35.268584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.268527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials" pod "69f122cc-8751-4b60-be36-ce4514ad9051" (UID: "69f122cc-8751-4b60-be36-ce4514ad9051"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:35.368775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.368743 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhq97\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-kube-api-access-bhq97\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:35.368775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.368771 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/69f122cc-8751-4b60-be36-ce4514ad9051-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:35.932583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.932530 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" podUID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" containerName="payload-processing" containerID="cri-o://937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e" gracePeriod=25 Oct 30 07:44:35.989351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.989314 5894 generic.go:356] "Generic (PLEG): container finished" podID="69f122cc-8751-4b60-be36-ce4514ad9051" containerID="5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0" exitCode=0 Oct 30 07:44:35.989534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.989361 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" Oct 30 07:44:35.989534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.989396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" event={"ID":"69f122cc-8751-4b60-be36-ce4514ad9051","Type":"ContainerDied","Data":"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0"} Oct 30 07:44:35.989534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.989452 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq" event={"ID":"69f122cc-8751-4b60-be36-ce4514ad9051","Type":"ContainerDied","Data":"d5738db2be45aa2a12fd8dff8215c28ff6937a482b017044b1b129f803b5fb95"} Oct 30 07:44:35.989534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:35.989477 5894 scope.go:122] "RemoveContainer" containerID="5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0" Oct 30 07:44:36.045875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.045846 5894 scope.go:122] "RemoveContainer" containerID="5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0" Oct 30 07:44:36.046261 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:36.046231 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0\": container with ID starting with 5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0 not found: ID does not exist" containerID="5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0" Oct 30 07:44:36.046381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.046272 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0"} err="failed to get container status \"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0\": rpc error: code = NotFound desc = could not find container \"5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0\": container with ID starting with 5353437deea6484e8057092e45e7592a17c57553b9c10cceb2d113fbc79228f0 not found: ID does not exist" Oct 30 07:44:36.072460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.072428 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:44:36.086029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.085971 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-7995nmkvq"] Oct 30 07:44:36.217857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.217825 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:36.282676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.282609 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\") pod \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " Oct 30 07:44:36.284249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.284212 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume" pod "a4c936f7-6dff-46e9-8b82-4415054e8aa4" (UID: "a4c936f7-6dff-46e9-8b82-4415054e8aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:36.384992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.384961 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\") pod \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " Oct 30 07:44:36.385264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.385084 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\") pod \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " Oct 30 07:44:36.385264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.385112 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\") pod \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\" (UID: \"a4c936f7-6dff-46e9-8b82-4415054e8aa4\") " Oct 30 07:44:36.385416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.385378 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume" pod "a4c936f7-6dff-46e9-8b82-4415054e8aa4" (UID: "a4c936f7-6dff-46e9-8b82-4415054e8aa4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:36.385781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.385755 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:36.385875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.385782 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a4c936f7-6dff-46e9-8b82-4415054e8aa4-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:36.387347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.387323 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5" pod "a4c936f7-6dff-46e9-8b82-4415054e8aa4" (UID: "a4c936f7-6dff-46e9-8b82-4415054e8aa4"). InnerVolumeSpecName "kube-api-access-22sp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:36.387436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.387357 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials" pod "a4c936f7-6dff-46e9-8b82-4415054e8aa4" (UID: "a4c936f7-6dff-46e9-8b82-4415054e8aa4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:36.433533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.433496 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f122cc-8751-4b60-be36-ce4514ad9051" path="/var/lib/kubelet/pods/69f122cc-8751-4b60-be36-ce4514ad9051/volumes" Oct 30 07:44:36.487338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.487255 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22sp5\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-kube-api-access-22sp5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:36.487338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.487287 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a4c936f7-6dff-46e9-8b82-4415054e8aa4-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:36.545066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:36.545029 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:37.001436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.001399 5894 generic.go:356] "Generic (PLEG): container finished" podID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" containerID="937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e" exitCode=0 Oct 30 07:44:37.001937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.001452 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" Oct 30 07:44:37.001937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.001475 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" event={"ID":"a4c936f7-6dff-46e9-8b82-4415054e8aa4","Type":"ContainerDied","Data":"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e"} Oct 30 07:44:37.001937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.001531 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg" event={"ID":"a4c936f7-6dff-46e9-8b82-4415054e8aa4","Type":"ContainerDied","Data":"a7fe89702331e9c4ac114aee0ac1da38124bdc88850787d8eb4a7c4ab0b50f1e"} Oct 30 07:44:37.001937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.001553 5894 scope.go:122] "RemoveContainer" containerID="937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e" Oct 30 07:44:37.013341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.013321 5894 scope.go:122] "RemoveContainer" containerID="937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e" Oct 30 07:44:37.013659 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:37.013628 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e\": container with ID starting with 937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e not found: ID does not exist" containerID="937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e" Oct 30 07:44:37.013730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.013667 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e"} err="failed to get container status \"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e\": rpc error: code = NotFound desc = could not find container \"937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e\": container with ID starting with 937546b1b8b50f64ca784afd89bb7ab9aa5f31b687a727e2164a6f4c7f913c7e not found: ID does not exist" Oct 30 07:44:37.065167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.065138 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:37.077372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.077328 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6051b2/payload-processing-external-model-e2e-extmod-a-6051b2-c65bwhzvg"] Oct 30 07:44:37.439242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.439205 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:37.567884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.567853 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:37.699969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:37.699883 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:38.220246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:38.220208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:38.433696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:38.433653 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" path="/var/lib/kubelet/pods/a4c936f7-6dff-46e9-8b82-4415054e8aa4/volumes" Oct 30 07:44:39.946285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:39.946250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:40.182319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:40.182282 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:44:40.273195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:40.273095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:40.538823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:40.538785 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:40.885544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:40.885457 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:41.228043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:41.227951 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.001954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.001921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.160225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.160184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.255164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255077 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw"] Oct 30 07:44:42.255939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255922 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7e60fc68-daa8-4499-8586-5ca3c27e3771" containerName="revoke-keys" Oct 30 07:44:42.255939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255941 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e60fc68-daa8-4499-8586-5ca3c27e3771" containerName="revoke-keys" Oct 30 07:44:42.256043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255960 5894 cpu_manager.go:414] "containerMap: removing container" podUID="69f122cc-8751-4b60-be36-ce4514ad9051" containerName="payload-processing" Oct 30 07:44:42.256043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255966 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f122cc-8751-4b60-be36-ce4514ad9051" containerName="payload-processing" Oct 30 07:44:42.256043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255979 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" containerName="payload-processing" Oct 30 07:44:42.256043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.255985 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" containerName="payload-processing" Oct 30 07:44:42.256174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.256110 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e60fc68-daa8-4499-8586-5ca3c27e3771" containerName="revoke-keys" Oct 30 07:44:42.256174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.256121 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f122cc-8751-4b60-be36-ce4514ad9051" containerName="payload-processing" Oct 30 07:44:42.256174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.256129 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c936f7-6dff-46e9-8b82-4415054e8aa4" containerName="payload-processing" Oct 30 07:44:42.261349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.261329 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.281944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.281877 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw"] Oct 30 07:44:42.287315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.287268 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.288080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.288054 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"agc-cred-aff03d9276a9\"" Oct 30 07:44:42.288223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.288175 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-e2e--f9d83428-dockercfg-pkzt8\"" Oct 30 07:44:42.300347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.300319 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.300500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.300440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.300574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.300522 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.300668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.300622 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs266\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.426658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.405545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.426981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.406521 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.427154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.427132 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.427250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.427231 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.427338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.427281 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs266\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.427847 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.427822 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:42.427847 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.427846 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:42.428048 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.427920 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials podName:09d358aa-e44d-47a2-bb30-7aa1e4a8b1af nodeName:}" failed. No retries permitted until 2026-10-30 07:44:42.927899207 +0000 UTC m=+2846.618816637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:42.428825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.428800 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.489659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.477365 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs266\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.651056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.651018 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw"] Oct 30 07:44:42.651565 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.651541 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" podUID="09d358aa-e44d-47a2-bb30-7aa1e4a8b1af" Oct 30 07:44:42.739548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.739504 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.852601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.852546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:42.935000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:42.934916 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:42.935188 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.935067 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:42.935188 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.935088 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:42.935188 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:42.935143 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials podName:09d358aa-e44d-47a2-bb30-7aa1e4a8b1af nodeName:}" failed. No retries permitted until 2026-10-30 07:44:43.935127442 +0000 UTC m=+2847.626044878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:43.130326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.129509 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:43.143453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.143424 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:43.244459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.244372 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\") pod \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " Oct 30 07:44:43.244628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.244470 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\") pod \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " Oct 30 07:44:43.244628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.244506 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\") pod \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " Oct 30 07:44:43.244882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.244855 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume" pod "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:43.244882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.244875 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume" pod "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:43.245150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.245130 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:43.245222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.245152 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:43.246925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.246896 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266" pod "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af"). InnerVolumeSpecName "kube-api-access-bs266". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:43.347163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.347128 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs266\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-kube-api-access-bs266\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:43.901886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.901788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:43.964656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:43.962971 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw\" (UID: \"09d358aa-e44d-47a2-bb30-7aa1e4a8b1af\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:43.964656 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:43.963195 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-b096ec/agc-cred-aff03d9276a9: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:43.964656 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:43.963215 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw: secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:43.964656 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:43.963292 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials podName:09d358aa-e44d-47a2-bb30-7aa1e4a8b1af nodeName:}" failed. No retries permitted until 2026-10-30 07:44:45.963271103 +0000 UTC m=+2849.654188541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" (UID: "09d358aa-e44d-47a2-bb30-7aa1e4a8b1af") : secret "agc-cred-aff03d9276a9" not found Oct 30 07:44:44.137433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.137391 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw" Oct 30 07:44:44.224803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.224716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:44.234763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.234731 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw"] Oct 30 07:44:44.239117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.239088 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-55d422zqw"] Oct 30 07:44:44.353779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.353737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:44.381272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.381240 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:44.432999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.432968 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d358aa-e44d-47a2-bb30-7aa1e4a8b1af" path="/var/lib/kubelet/pods/09d358aa-e44d-47a2-bb30-7aa1e4a8b1af/volumes" Oct 30 07:44:44.726232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:44.726193 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:45.183143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.183096 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" podUID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" containerName="payload-processing" containerID="cri-o://2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431" gracePeriod=25 Oct 30 07:44:45.200174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.200134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:45.311402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.311363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:45.715439 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.715402 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:44:45.716052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.716013 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:45.817469 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.817422 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\") pod \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " Oct 30 07:44:45.817733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.817569 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\") pod \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " Oct 30 07:44:45.817733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.817673 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\") pod \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " Oct 30 07:44:45.817885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.817839 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\") pod \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\" (UID: \"d1e7c955-d473-4bbb-9100-4ce3604e10a9\") " Oct 30 07:44:45.818096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.818067 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume" pod "d1e7c955-d473-4bbb-9100-4ce3604e10a9" (UID: "d1e7c955-d473-4bbb-9100-4ce3604e10a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:45.818202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.818074 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume" pod "d1e7c955-d473-4bbb-9100-4ce3604e10a9" (UID: "d1e7c955-d473-4bbb-9100-4ce3604e10a9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:44:45.818563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.818539 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:45.818726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.818566 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1e7c955-d473-4bbb-9100-4ce3604e10a9-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:45.821137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.820971 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc" pod "d1e7c955-d473-4bbb-9100-4ce3604e10a9" (UID: "d1e7c955-d473-4bbb-9100-4ce3604e10a9"). InnerVolumeSpecName "kube-api-access-6q6lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:45.821137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.821096 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials" pod "d1e7c955-d473-4bbb-9100-4ce3604e10a9" (UID: "d1e7c955-d473-4bbb-9100-4ce3604e10a9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:44:45.920065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.919972 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:45.920065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:45.920003 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6lc\" (UniqueName: \"kubernetes.io/projected/d1e7c955-d473-4bbb-9100-4ce3604e10a9-kube-api-access-6q6lc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:44:46.167270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.167231 5894 generic.go:356] "Generic (PLEG): container finished" podID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" containerID="2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431" exitCode=0 Oct 30 07:44:46.167496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.167282 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" Oct 30 07:44:46.167496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.167310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" event={"ID":"d1e7c955-d473-4bbb-9100-4ce3604e10a9","Type":"ContainerDied","Data":"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431"} Oct 30 07:44:46.167496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.167373 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5" event={"ID":"d1e7c955-d473-4bbb-9100-4ce3604e10a9","Type":"ContainerDied","Data":"90f2aac9b0f1ddc655d2508f8773183520bad8c6c780e1f2ad95fd724e7ac1fb"} Oct 30 07:44:46.167496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.167397 5894 scope.go:122] "RemoveContainer" containerID="2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431" Oct 30 07:44:46.181523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.181500 5894 scope.go:122] "RemoveContainer" containerID="2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431" Oct 30 07:44:46.181942 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:44:46.181916 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431\": container with ID starting with 2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431 not found: ID does not exist" containerID="2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431" Oct 30 07:44:46.182050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.181953 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431"} err="failed to get container status \"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431\": rpc error: code = NotFound desc = could not find container \"2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431\": container with ID starting with 2535ff73134759abf307ecdc0758565dc7151bfcbf85f3be153a89b53633e431 not found: ID does not exist" Oct 30 07:44:46.261584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.261546 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:44:46.292102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.292065 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-cbc8qpxr5"] Oct 30 07:44:46.436903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:46.436827 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" path="/var/lib/kubelet/pods/d1e7c955-d473-4bbb-9100-4ce3604e10a9/volumes" Oct 30 07:44:47.902112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:47.902062 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:48.926326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:48.926290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:49.083242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:49.083207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:49.701015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:49.700979 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:49.831644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:49.831592 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:49.952391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:49.952300 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:50.318764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.318724 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:50.458886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.458849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:50.623167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.623071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:50.747931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.747898 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:50.761403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.761370 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:44:50.762249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.762229 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" containerName="payload-processing" Oct 30 07:44:50.762307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.762251 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" containerName="payload-processing" Oct 30 07:44:50.762418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.762407 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e7c955-d473-4bbb-9100-4ce3604e10a9" containerName="payload-processing" Oct 30 07:44:50.768095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.768070 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:50.786326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.786294 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:44:50.792985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.792952 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fae1d2\"" Oct 30 07:44:50.803026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.802991 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:44:50.808419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.808393 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:50.829910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.829863 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fae1d2-dockercfg-2jwsq\"" Oct 30 07:44:50.831805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.831770 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:44:50.930583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.908976 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kn7t\" (UniqueName: \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:50.931913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:50.931878 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.037660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.037059 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpgt\" (UniqueName: \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.037660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.037125 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.037660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.037194 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kn7t\" (UniqueName: \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.037660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.037284 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.038350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.038093 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.075072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.075036 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kn7t\" (UniqueName: \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\") pod \"payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.085589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.085551 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:51.138298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.138072 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpgt\" (UniqueName: \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.138298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.138122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.139099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.139075 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.167264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.167196 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpgt\" (UniqueName: \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\") pod \"payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.430831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.430803 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:51.432327 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:51.432299 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0c4c56_76ef_4a02_a27f_52033755b4a2.slice/crio-f8102ef55be81723304702ced50bfdc27a4e5246c3d9eb63e84daf2255b1911b WatchSource:0}: Error finding container f8102ef55be81723304702ced50bfdc27a4e5246c3d9eb63e84daf2255b1911b: Status 404 returned error can't find the container with id f8102ef55be81723304702ced50bfdc27a4e5246c3d9eb63e84daf2255b1911b Oct 30 07:44:51.435545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.435518 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:44:51.594430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.594393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:51.749063 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:44:51.749025 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2d78cd_f433_4591_ad91_28935165fe2f.slice/crio-320c4bc88f5a15477ee16114f96a5506786a8515d2f70ec01dd1e67178aef042 WatchSource:0}: Error finding container 320c4bc88f5a15477ee16114f96a5506786a8515d2f70ec01dd1e67178aef042: Status 404 returned error can't find the container with id 320c4bc88f5a15477ee16114f96a5506786a8515d2f70ec01dd1e67178aef042 Oct 30 07:44:51.750273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:51.750250 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:44:52.223979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.223940 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" event={"ID":"8e2d78cd-f433-4591-ad91-28935165fe2f","Type":"ContainerStarted","Data":"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3"} Oct 30 07:44:52.224482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.224036 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" event={"ID":"8e2d78cd-f433-4591-ad91-28935165fe2f","Type":"ContainerStarted","Data":"320c4bc88f5a15477ee16114f96a5506786a8515d2f70ec01dd1e67178aef042"} Oct 30 07:44:52.224482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.224179 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:44:52.226089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.226060 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" event={"ID":"dd0c4c56-76ef-4a02-a27f-52033755b4a2","Type":"ContainerStarted","Data":"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13"} Oct 30 07:44:52.226232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.226099 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" event={"ID":"dd0c4c56-76ef-4a02-a27f-52033755b4a2","Type":"ContainerStarted","Data":"f8102ef55be81723304702ced50bfdc27a4e5246c3d9eb63e84daf2255b1911b"} Oct 30 07:44:52.226232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.226139 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:44:52.268554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.268504 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" podStartSLOduration=2.268490206 podStartE2EDuration="2.268490206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:52.265347713 +0000 UTC m=+2855.956265151" watchObservedRunningTime="2026-10-30 07:44:52.268490206 +0000 UTC m=+2855.959407651" Oct 30 07:44:52.308822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:52.308768 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" podStartSLOduration=2.308754898 podStartE2EDuration="2.308754898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:44:52.307174878 +0000 UTC m=+2855.998092315" watchObservedRunningTime="2026-10-30 07:44:52.308754898 +0000 UTC m=+2855.999672418" Oct 30 07:44:53.408762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:53.408723 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:55.030934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:55.030891 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.187967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.187931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.344811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.344773 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.473361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.473266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.676018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.675974 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.797853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.797818 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:56.970833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:56.970786 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:57.617812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:57.617777 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:57.736986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:57.736937 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:44:58.745802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:44:58.745771 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:00.178416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.178379 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:00.183700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.183653 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:00.183886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.183723 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.188673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.188618 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:00.188841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.188773 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.193087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.193065 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:00.193208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.193190 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.197785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.197767 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.209144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.209119 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:45:00.209544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.209490 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:45:00.211347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.211113 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:00.227654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.215401 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:45:00.231727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.230101 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.231727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.231380 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:00.238629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.238607 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:00.245865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.245839 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256304 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt48n\" (UniqueName: \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256362 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256418 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256465 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256663 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmn8\" (UniqueName: \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256703 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnq7\" (UniqueName: \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256826 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.256860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.256860 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9p4p\" (UniqueName: \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.266382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.262002 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:00.287404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.287360 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:00.302704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.300403 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:00.321341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.321311 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:00.321555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.321460 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.327302 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.327276 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:00.328518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.328485 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.336000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.335974 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:00.340492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.339897 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.354571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.354543 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:00.354894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.354792 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359106 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359172 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359215 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359381 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmn8\" (UniqueName: \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359415 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnq7\" (UniqueName: \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359466 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqxj\" (UniqueName: \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359550 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359587 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359619 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9p4p\" (UniqueName: \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.360372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.359717 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt48n\" (UniqueName: \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.360815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.360723 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.363562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.363484 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.364118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.364094 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.364317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.364291 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.367099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.367068 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:00.367213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.367115 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:00.368624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.368440 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.373295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.373273 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:00.373457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.373314 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:00.374579 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.374560 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.385765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.381730 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.385765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.381628 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:00.407682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.400807 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:00.407682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.401309 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.407682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.406324 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt48n\" (UniqueName: \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.407682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.407562 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.408031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.407556 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnq7\" (UniqueName: \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.408784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.408735 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:00.412687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.411010 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:00.413305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.413279 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmn8\" (UniqueName: \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.413438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.413344 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9p4p\" (UniqueName: \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.427694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.424283 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503081 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503149 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503186 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503274 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs6kr\" (UniqueName: \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503352 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7ftc\" (UniqueName: \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503387 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hcm8\" (UniqueName: \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503424 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503459 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503490 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-279xt\" (UniqueName: \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503525 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9lw\" (UniqueName: \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503560 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503795 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtn7t\" (UniqueName: \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503862 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503920 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzdr\" (UniqueName: \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503944 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkqzf\" (UniqueName: \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.503988 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.504022 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.504052 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqxj\" (UniqueName: \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.504087 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.504118 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6tj9\" (UniqueName: \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.505025 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.523714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.505782 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:00.534988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.532053 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:00.562246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.562207 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:00.569687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.569654 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:00.571751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.571444 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:00.571751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.571519 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:00.571751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.571546 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:00.571751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.571570 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605156 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtn7t\" (UniqueName: \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605214 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605263 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzdr\" (UniqueName: \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605288 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkqzf\" (UniqueName: \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605326 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605356 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605388 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605419 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6tj9\" (UniqueName: \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605499 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605535 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.606763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.605616 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs6kr\" (UniqueName: \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607062 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7ftc\" (UniqueName: \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607114 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hcm8\" (UniqueName: \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607237 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607275 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607302 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-279xt\" (UniqueName: \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607419 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9lw\" (UniqueName: \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.607547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.607472 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.612852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.608563 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.612852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.610328 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqxj\" (UniqueName: \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.612852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.611151 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.614745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.613720 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.614745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.614588 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.616162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.615341 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.616162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.615921 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.616162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.616116 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.618440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.617948 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.619418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.619392 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.623680 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.621820 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:00.630985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.630930 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:00.631484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.631096 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.646744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.646027 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkqzf\" (UniqueName: \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.648360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.648217 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:00.649214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.649063 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.653214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.653186 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-279xt\" (UniqueName: \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.653577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.653558 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:00.656974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.656729 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:00.664171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.664148 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.679591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.679565 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs6kr\" (UniqueName: \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\") pod \"maas-api-key-cleanup-29889105-r58rq\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.680056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.680025 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9lw\" (UniqueName: \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\") pod \"maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.681155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.680864 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6tj9\" (UniqueName: \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.683332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.683017 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtn7t\" (UniqueName: \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.683332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.683148 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hcm8\" (UniqueName: \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.687985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.687949 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzdr\" (UniqueName: \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.688251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.688231 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7ftc\" (UniqueName: \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.688900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.688880 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:00.689107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.689082 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:00.689215 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.689201 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:00.695287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.693057 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:00.695287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.694516 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:00.714882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.709366 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:00.714882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.713490 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:00.728891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.725085 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:00.776116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.776051 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:00.777542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.777510 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:00.777795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.777613 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:00.777795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.777624 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:00.778223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.778201 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:00.783244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.783020 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:00.811137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.809404 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.812127 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813456 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l2zm\" (UniqueName: \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813510 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp45l\" (UniqueName: \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813554 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813818 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.815430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.813877 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.826539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.826232 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"openshift-service-ca.crt\"" Oct 30 07:45:00.826539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.826360 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"routing-overlay\"" Oct 30 07:45:00.826539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.826419 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"kube-root-ca.crt\"" Oct 30 07:45:00.826539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.826497 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"agc-cred-7dccf7a707b2\"" Oct 30 07:45:00.864546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.864488 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:00.894045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.891138 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:00.929284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.929253 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-b096ec\"/\"payload-processing-external-model-plugins-e2e-extmod-a-b096ec\"" Oct 30 07:45:00.930090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.929862 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:00.931075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931043 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.931173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931112 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:00.931173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931148 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.931295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931247 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlvks\" (UniqueName: \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:00.931295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931287 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfrg\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:00.931408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931355 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.931408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931387 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:00.931511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931442 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l2zm\" (UniqueName: \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.931511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931488 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:00.931617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931514 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp45l\" (UniqueName: \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.931617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931554 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.932044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.931656 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:00.933393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.932663 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:00.933393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.932749 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:00.934819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.934791 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:00.968917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:00.968788 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:01.034772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.034082 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfrg\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.034772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.034200 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.034772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.034312 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.034772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.034408 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.034772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.034622 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:01.037044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.036932 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.073034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.037295 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlvks\" (UniqueName: \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:01.073759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.073255 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.074990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.074558 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:01.127914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.127613 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:01.264718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.249490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:01.264718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.254021 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l2zm\" (UniqueName: \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\") pod \"maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:01.277114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.277073 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:01.294562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.294456 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp45l\" (UniqueName: \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:01.333229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.330668 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.334551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.333871 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlvks\" (UniqueName: \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:01.335001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.334976 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfrg\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\") pod \"payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.398001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.397930 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerStarted","Data":"cb57790680d2b751d47de927a14d0196b5212dda4fbb733364c3a6af52c1926f"} Oct 30 07:45:01.448887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.447969 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:01.487186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.487159 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:01.508062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.508025 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:01.525179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.524960 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:01.648457 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:01.648414 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1666912_5749_4286_bb70_75f9968f3cca.slice/crio-7fee7612e49c008f8fa3ff2ee5fc91933451e973de5509d48aba86fb0484d181 WatchSource:0}: Error finding container 7fee7612e49c008f8fa3ff2ee5fc91933451e973de5509d48aba86fb0484d181: Status 404 returned error can't find the container with id 7fee7612e49c008f8fa3ff2ee5fc91933451e973de5509d48aba86fb0484d181 Oct 30 07:45:01.652601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.652529 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:01.816373 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:01.816337 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5125cf24_506e_47c7_85f9_ff51fa746129.slice/crio-da463eaedf585a23af11656af0616ebedf90541103df67714fb372e8c358ffc1 WatchSource:0}: Error finding container da463eaedf585a23af11656af0616ebedf90541103df67714fb372e8c358ffc1: Status 404 returned error can't find the container with id da463eaedf585a23af11656af0616ebedf90541103df67714fb372e8c358ffc1 Oct 30 07:45:01.816692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:01.816665 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:02.149330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.149153 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:02.159525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.159490 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:02.176754 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:02.176716 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89601e6b_88f3_4a7a_9d88_cc02c7c9b67c.slice/crio-cb1fd08098efd1d314c7bf2a887c604eca0335b4f6cae8c77b2b898a0bd1d55b WatchSource:0}: Error finding container cb1fd08098efd1d314c7bf2a887c604eca0335b4f6cae8c77b2b898a0bd1d55b: Status 404 returned error can't find the container with id cb1fd08098efd1d314c7bf2a887c604eca0335b4f6cae8c77b2b898a0bd1d55b Oct 30 07:45:02.183865 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:02.183281 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc895d4bf_e5f2_44cb_959f_d36c5b14e623.slice/crio-2b069415c057d97a82c65d776fd0fdf047b076f53e9e1f2f6c0f805492d86e04 WatchSource:0}: Error finding container 2b069415c057d97a82c65d776fd0fdf047b076f53e9e1f2f6c0f805492d86e04: Status 404 returned error can't find the container with id 2b069415c057d97a82c65d776fd0fdf047b076f53e9e1f2f6c0f805492d86e04 Oct 30 07:45:02.191958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.191885 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:02.217282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.217248 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:02.223421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.223237 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:02.229079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.229046 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:02.247114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.244969 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:02.252856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.252828 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:02.259807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.259553 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:02.276394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.276315 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:02.286700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.283940 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:02.295401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.295369 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:02.306376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.301812 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:02.315618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.315588 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:02.378832 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:02.378611 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22679da9_0561_4760_a07b_6adff3f83ef0.slice/crio-9811c3696453d3137b3adf8ac1e516585256e80baadb02b2d879994aafcf6b3d WatchSource:0}: Error finding container 9811c3696453d3137b3adf8ac1e516585256e80baadb02b2d879994aafcf6b3d: Status 404 returned error can't find the container with id 9811c3696453d3137b3adf8ac1e516585256e80baadb02b2d879994aafcf6b3d Oct 30 07:45:02.407921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.407877 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerStarted","Data":"46f1bb996e5500e245e2a47196a7945342667e64510d0238830d6d787167bc2a"} Oct 30 07:45:02.409443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.409411 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerStarted","Data":"64fb286251fe2e0276baf3e05ff20e67819b52138edc98d0713b16fa8faa2b7f"} Oct 30 07:45:02.411121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.411096 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerStarted","Data":"99a7a32d8a9dd6494bb18c37313391e11088000c94f91db39c3edec22b6d5827"} Oct 30 07:45:02.412890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.412867 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerStarted","Data":"ee69d8ac204da8981763a43aa751dacfdb323ba59fb9aeb2f8b30b2f04440132"} Oct 30 07:45:02.412991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.412912 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerStarted","Data":"7fee7612e49c008f8fa3ff2ee5fc91933451e973de5509d48aba86fb0484d181"} Oct 30 07:45:02.414291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.414270 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerStarted","Data":"e382765d1fd3708c3d811327e444b3322dcc47a0ee30390948b1b98cea53a271"} Oct 30 07:45:02.415586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.415562 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerStarted","Data":"ebb30a807328ddd4a39e450a64755b23e80fe80dcbadae528696ea70a11b597d"} Oct 30 07:45:02.416996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.416974 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerStarted","Data":"cd152051d063f848c099088087f2c32d008a2d752a93c244539164f794136539"} Oct 30 07:45:02.418273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.418255 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerStarted","Data":"cb1fd08098efd1d314c7bf2a887c604eca0335b4f6cae8c77b2b898a0bd1d55b"} Oct 30 07:45:02.419507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.419481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerStarted","Data":"9811c3696453d3137b3adf8ac1e516585256e80baadb02b2d879994aafcf6b3d"} Oct 30 07:45:02.420620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.420602 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerStarted","Data":"2b069415c057d97a82c65d776fd0fdf047b076f53e9e1f2f6c0f805492d86e04"} Oct 30 07:45:02.421791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.421766 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerStarted","Data":"3d52c76763951dbe16a9d2c508e83dde5961a1bdf879fccf7afd1bf100f21f34"} Oct 30 07:45:02.423075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.423056 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" event={"ID":"6e4cceaa-bc57-4862-8212-f6920f930785","Type":"ContainerStarted","Data":"3cadeb8db3868ed1ffa03dfb574439e87ca1a2b1ac1514da263e576fce5a3261"} Oct 30 07:45:02.436360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.436326 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerStarted","Data":"433b61d96cf4a980a4cd0ed9ef2fe5dccf0166d9bb299acb42afd4ba533bdeee"} Oct 30 07:45:02.436500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.436372 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerStarted","Data":"ffab1d50c965c98e8a83a00a517d1f32603e13f1a3951c97e4c30fdf2b97f4c2"} Oct 30 07:45:02.436500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.436394 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerStarted","Data":"da463eaedf585a23af11656af0616ebedf90541103df67714fb372e8c358ffc1"} Oct 30 07:45:02.436500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.436409 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerStarted","Data":"e486cd0acdf6dedda519afa9693347a02e931dac880613c779593a167a408e25"} Oct 30 07:45:02.436500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.436424 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerStarted","Data":"2e63c8eb3be48801fc38781e9b8aec373a9d0b959d91166f33ac253a62911ef4"} Oct 30 07:45:02.448520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.448496 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:02.452512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.452470 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:02.457544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.457486 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" podStartSLOduration=2.45746992 podStartE2EDuration="2.45746992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.456376266 +0000 UTC m=+2866.147293705" watchObservedRunningTime="2026-10-30 07:45:02.45746992 +0000 UTC m=+2866.148387357" Oct 30 07:45:02.513908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.511466 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" podStartSLOduration=2.511446834 podStartE2EDuration="2.511446834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.509513928 +0000 UTC m=+2866.200431367" watchObservedRunningTime="2026-10-30 07:45:02.511446834 +0000 UTC m=+2866.202364272" Oct 30 07:45:02.519565 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:02.519536 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024f2407_22e9_4298_ae1c_5c604dc4dd35.slice/crio-5cb185bb0467be1f15b6c8fd2dc9966aa8cd45ca169d65982be7968396bb725e WatchSource:0}: Error finding container 5cb185bb0467be1f15b6c8fd2dc9966aa8cd45ca169d65982be7968396bb725e: Status 404 returned error can't find the container with id 5cb185bb0467be1f15b6c8fd2dc9966aa8cd45ca169d65982be7968396bb725e Oct 30 07:45:02.539586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.539528 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:45:02.949908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:02.949810 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:03.236983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.236955 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:45:03.237124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.237089 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:45:03.287223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.287162 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" podStartSLOduration=3.287139589 podStartE2EDuration="3.287139589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:02.557280787 +0000 UTC m=+2866.248198231" watchObservedRunningTime="2026-10-30 07:45:03.287139589 +0000 UTC m=+2866.978057027" Oct 30 07:45:03.442724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.442682 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerStarted","Data":"e0827612ba5be56b1e990f228c75db6276ad3aa0374aa19b2dba7f544ad45277"} Oct 30 07:45:03.445021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.444988 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerStarted","Data":"699e2827890476df5b45480b46630bab177a2da0443acf95913d885d9c614952"} Oct 30 07:45:03.448750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.448330 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerStarted","Data":"a290f891af0510009473afbdf4ed0a205ccee5037a59130ef629569d2fdd8a80"} Oct 30 07:45:03.450544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.450462 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerStarted","Data":"141a33db9fb514e2dd53ad1d8d564bd7ad33b0220b004da42f68bd181600ede6"} Oct 30 07:45:03.452768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.452731 5894 generic.go:356] "Generic (PLEG): container finished" podID="5125cf24-506e-47c7-85f9-ff51fa746129" containerID="ffab1d50c965c98e8a83a00a517d1f32603e13f1a3951c97e4c30fdf2b97f4c2" exitCode=60 Oct 30 07:45:03.452931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.452782 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerDied","Data":"ffab1d50c965c98e8a83a00a517d1f32603e13f1a3951c97e4c30fdf2b97f4c2"} Oct 30 07:45:03.453085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.453056 5894 scope.go:122] "RemoveContainer" containerID="ffab1d50c965c98e8a83a00a517d1f32603e13f1a3951c97e4c30fdf2b97f4c2" Oct 30 07:45:03.456720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.456661 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" event={"ID":"024f2407-22e9-4298-ae1c-5c604dc4dd35","Type":"ContainerStarted","Data":"5cb185bb0467be1f15b6c8fd2dc9966aa8cd45ca169d65982be7968396bb725e"} Oct 30 07:45:03.458917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.458894 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerStarted","Data":"817135c67be0dd642e137e85babc2bd275a96f962c4ca30ca34f4bb6e2532bd9"} Oct 30 07:45:03.460840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.460817 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerStarted","Data":"4160472349a6fce042d1879a17baa4e0e031ffb85b817dbfc465f9d8332461d7"} Oct 30 07:45:03.462518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.462489 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" event={"ID":"1b3448b7-cb81-416d-9793-e4e719e638a1","Type":"ContainerStarted","Data":"e3bfde9be2bc95d506c1131f1b84466b942f45a73b127b463e77a0ddc62e186e"} Oct 30 07:45:03.462625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.462538 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" event={"ID":"1b3448b7-cb81-416d-9793-e4e719e638a1","Type":"ContainerStarted","Data":"7caec6f82abec68af51ee99513dd9c9291604570311579563ba31f48ecde6078"} Oct 30 07:45:03.462625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.462566 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:03.464134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.464113 5894 generic.go:356] "Generic (PLEG): container finished" podID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerID="99a7a32d8a9dd6494bb18c37313391e11088000c94f91db39c3edec22b6d5827" exitCode=60 Oct 30 07:45:03.464248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.464193 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerDied","Data":"99a7a32d8a9dd6494bb18c37313391e11088000c94f91db39c3edec22b6d5827"} Oct 30 07:45:03.464359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.464343 5894 scope.go:122] "RemoveContainer" containerID="99a7a32d8a9dd6494bb18c37313391e11088000c94f91db39c3edec22b6d5827" Oct 30 07:45:03.465894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.465859 5894 generic.go:356] "Generic (PLEG): container finished" podID="a1666912-5749-4286-bb70-75f9968f3cca" containerID="ee69d8ac204da8981763a43aa751dacfdb323ba59fb9aeb2f8b30b2f04440132" exitCode=60 Oct 30 07:45:03.465965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.465939 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerDied","Data":"ee69d8ac204da8981763a43aa751dacfdb323ba59fb9aeb2f8b30b2f04440132"} Oct 30 07:45:03.466162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.466142 5894 scope.go:122] "RemoveContainer" containerID="ee69d8ac204da8981763a43aa751dacfdb323ba59fb9aeb2f8b30b2f04440132" Oct 30 07:45:03.467473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.467453 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerStarted","Data":"70efa65f3b576616ed25082660453fb1915111a7d910805fb3a1d21292d7759c"} Oct 30 07:45:03.506714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.506660 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" podStartSLOduration=3.50662983 podStartE2EDuration="3.50662983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.504494727 +0000 UTC m=+2867.195412164" watchObservedRunningTime="2026-10-30 07:45:03.50662983 +0000 UTC m=+2867.197547306" Oct 30 07:45:03.526737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.526701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:03.595356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.595296 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" podStartSLOduration=3.595280441 podStartE2EDuration="3.595280441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.594057056 +0000 UTC m=+2867.284974493" watchObservedRunningTime="2026-10-30 07:45:03.595280441 +0000 UTC m=+2867.286197922" Oct 30 07:45:03.691674 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.686251 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" podStartSLOduration=3.6862315 podStartE2EDuration="3.6862315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.66200329 +0000 UTC m=+2867.352920728" watchObservedRunningTime="2026-10-30 07:45:03.6862315 +0000 UTC m=+2867.377148940" Oct 30 07:45:03.747551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.747477 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" podStartSLOduration=3.747455424 podStartE2EDuration="3.747455424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.713108047 +0000 UTC m=+2867.404025486" watchObservedRunningTime="2026-10-30 07:45:03.747455424 +0000 UTC m=+2867.438372862" Oct 30 07:45:03.880692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:03.875798 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:04.172890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.172815 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" podStartSLOduration=4.172792011 podStartE2EDuration="4.172792011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:03.79229989 +0000 UTC m=+2867.483217329" watchObservedRunningTime="2026-10-30 07:45:04.172792011 +0000 UTC m=+2867.863709474" Oct 30 07:45:04.358238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.331051 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384358 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"agc-cred-b42b42106144\"" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384504 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"payload-processing-external-model-e2e--b7f37f47-dockercfg-q6p22\"" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384610 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"kube-root-ca.crt\"" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384747 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"payload-processing-external-model-plugins-e2e-extmod-b-0f76b0\"" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384888 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"openshift-service-ca.crt\"" Oct 30 07:45:04.412657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.384982 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-0f76b0\"/\"routing-overlay\"" Oct 30 07:45:04.649008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.624324 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.649008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.624682 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjfv\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.649008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.624732 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.649008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.624803 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.649008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.639526 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerStarted","Data":"b16d49cc9835f82121170151fabb8fe44041ed4e3e9345c17028021232af7410"} Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.721895 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.736535 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjfv\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.736595 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.736700 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.736823 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.742740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.740944 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.749010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.747764 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.767771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.767699 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.778495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.775898 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" podStartSLOduration=4.775877679 podStartE2EDuration="4.775877679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.255554062 +0000 UTC m=+2867.946471503" watchObservedRunningTime="2026-10-30 07:45:04.775877679 +0000 UTC m=+2868.466795117" Oct 30 07:45:04.804706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.803432 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjfv\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\") pod \"payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.827716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.827669 5894 generic.go:356] "Generic (PLEG): container finished" podID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerID="699e2827890476df5b45480b46630bab177a2da0443acf95913d885d9c614952" exitCode=60 Oct 30 07:45:04.827977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.827953 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerDied","Data":"699e2827890476df5b45480b46630bab177a2da0443acf95913d885d9c614952"} Oct 30 07:45:04.828676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.828615 5894 scope.go:122] "RemoveContainer" containerID="699e2827890476df5b45480b46630bab177a2da0443acf95913d885d9c614952" Oct 30 07:45:04.830139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.829978 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" podStartSLOduration=4.829964942 podStartE2EDuration="4.829964942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.385217813 +0000 UTC m=+2868.076135252" watchObservedRunningTime="2026-10-30 07:45:04.829964942 +0000 UTC m=+2868.520882379" Oct 30 07:45:04.851012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.846107 5894 scope.go:122] "RemoveContainer" containerID="ba8643d919e4d8185629f13e84ac585db88f69f4b52beb0ed3143a2bdd3e4d22" Oct 30 07:45:04.852749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.852692 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" podStartSLOduration=4.852672951 podStartE2EDuration="4.852672951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.742060704 +0000 UTC m=+2868.432978147" watchObservedRunningTime="2026-10-30 07:45:04.852672951 +0000 UTC m=+2868.543590389" Oct 30 07:45:04.863430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.863377 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" podStartSLOduration=4.863358731 podStartE2EDuration="4.863358731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.790258788 +0000 UTC m=+2868.481176256" watchObservedRunningTime="2026-10-30 07:45:04.863358731 +0000 UTC m=+2868.554276169" Oct 30 07:45:04.880370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.880198 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" podStartSLOduration=4.88018064 podStartE2EDuration="4.88018064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.842566606 +0000 UTC m=+2868.533484046" watchObservedRunningTime="2026-10-30 07:45:04.88018064 +0000 UTC m=+2868.571098078" Oct 30 07:45:04.883199 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:04.883160 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:45:04.892961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.892294 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" podStartSLOduration=4.892199336 podStartE2EDuration="4.892199336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.889048448 +0000 UTC m=+2868.579965886" watchObservedRunningTime="2026-10-30 07:45:04.892199336 +0000 UTC m=+2868.583116776" Oct 30 07:45:04.935059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.935032 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:04.979280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:04.979211 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" podStartSLOduration=4.979189524 podStartE2EDuration="4.979189524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:04.973987472 +0000 UTC m=+2868.664904926" watchObservedRunningTime="2026-10-30 07:45:04.979189524 +0000 UTC m=+2868.670106962" Oct 30 07:45:05.017295 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.017238 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" podStartSLOduration=5.017220117 podStartE2EDuration="5.017220117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:05.014711682 +0000 UTC m=+2868.705629118" watchObservedRunningTime="2026-10-30 07:45:05.017220117 +0000 UTC m=+2868.708137556" Oct 30 07:45:05.253484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.253447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:05.360203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.360166 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:05.374356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.374314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:05.393003 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:05.392957 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085812ea_9161_4900_8bde_4c5c585d513f.slice/crio-7065acfec8954fe7e75b5f0b5af91bdc1154a857b1eb2a339ee3219f3e8b7c20 WatchSource:0}: Error finding container 7065acfec8954fe7e75b5f0b5af91bdc1154a857b1eb2a339ee3219f3e8b7c20: Status 404 returned error can't find the container with id 7065acfec8954fe7e75b5f0b5af91bdc1154a857b1eb2a339ee3219f3e8b7c20 Oct 30 07:45:05.865315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.865284 5894 generic.go:356] "Generic (PLEG): container finished" podID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerID="b16d49cc9835f82121170151fabb8fe44041ed4e3e9345c17028021232af7410" exitCode=60 Oct 30 07:45:05.865526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.865369 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerDied","Data":"b16d49cc9835f82121170151fabb8fe44041ed4e3e9345c17028021232af7410"} Oct 30 07:45:05.865644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.865616 5894 scope.go:122] "RemoveContainer" containerID="b16d49cc9835f82121170151fabb8fe44041ed4e3e9345c17028021232af7410" Oct 30 07:45:05.867498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.867418 5894 generic.go:356] "Generic (PLEG): container finished" podID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerID="4160472349a6fce042d1879a17baa4e0e031ffb85b817dbfc465f9d8332461d7" exitCode=60 Oct 30 07:45:05.867498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.867459 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerDied","Data":"4160472349a6fce042d1879a17baa4e0e031ffb85b817dbfc465f9d8332461d7"} Oct 30 07:45:05.868907 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.868887 5894 scope.go:122] "RemoveContainer" containerID="4160472349a6fce042d1879a17baa4e0e031ffb85b817dbfc465f9d8332461d7" Oct 30 07:45:05.870006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.869986 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerStarted","Data":"2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50"} Oct 30 07:45:05.872588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.872562 5894 generic.go:356] "Generic (PLEG): container finished" podID="a1666912-5749-4286-bb70-75f9968f3cca" containerID="a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca" exitCode=60 Oct 30 07:45:05.872756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.872686 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerDied","Data":"a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca"} Oct 30 07:45:05.872756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.872733 5894 scope.go:122] "RemoveContainer" containerID="ee69d8ac204da8981763a43aa751dacfdb323ba59fb9aeb2f8b30b2f04440132" Oct 30 07:45:05.872987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.872967 5894 scope.go:122] "RemoveContainer" containerID="a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca" Oct 30 07:45:05.873370 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:05.873337 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj_odh-ai-gateway-infra(a1666912-5749-4286-bb70-75f9968f3cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" podUID="a1666912-5749-4286-bb70-75f9968f3cca" Oct 30 07:45:05.875223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.875203 5894 generic.go:356] "Generic (PLEG): container finished" podID="1de640e8-db88-438e-8279-e7efd370deb6" containerID="70efa65f3b576616ed25082660453fb1915111a7d910805fb3a1d21292d7759c" exitCode=60 Oct 30 07:45:05.875315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.875276 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerDied","Data":"70efa65f3b576616ed25082660453fb1915111a7d910805fb3a1d21292d7759c"} Oct 30 07:45:05.875627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.875608 5894 scope.go:122] "RemoveContainer" containerID="70efa65f3b576616ed25082660453fb1915111a7d910805fb3a1d21292d7759c" Oct 30 07:45:05.877464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.877443 5894 generic.go:356] "Generic (PLEG): container finished" podID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerID="1ce9d064559f8afb2a457ad2c0a6d4b7c973eff1869798f5e6776a2e25ed69a6" exitCode=60 Oct 30 07:45:05.877572 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.877474 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerDied","Data":"1ce9d064559f8afb2a457ad2c0a6d4b7c973eff1869798f5e6776a2e25ed69a6"} Oct 30 07:45:05.878114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.877904 5894 scope.go:122] "RemoveContainer" containerID="1ce9d064559f8afb2a457ad2c0a6d4b7c973eff1869798f5e6776a2e25ed69a6" Oct 30 07:45:05.880234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.880207 5894 generic.go:356] "Generic (PLEG): container finished" podID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerID="23268e8c1bc72c7af8f2ef5b66c3083c93e8b2bbde8b5987f29b64c0590cb87e" exitCode=60 Oct 30 07:45:05.880340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.880297 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerDied","Data":"23268e8c1bc72c7af8f2ef5b66c3083c93e8b2bbde8b5987f29b64c0590cb87e"} Oct 30 07:45:05.880721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.880621 5894 scope.go:122] "RemoveContainer" containerID="23268e8c1bc72c7af8f2ef5b66c3083c93e8b2bbde8b5987f29b64c0590cb87e" Oct 30 07:45:05.884107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.884082 5894 generic.go:356] "Generic (PLEG): container finished" podID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerID="3d3452aa7ed632c71b41a89cdcb30a684c3577c11e08ecedad9676689947660e" exitCode=60 Oct 30 07:45:05.884224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.884171 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerDied","Data":"3d3452aa7ed632c71b41a89cdcb30a684c3577c11e08ecedad9676689947660e"} Oct 30 07:45:05.884508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.884488 5894 scope.go:122] "RemoveContainer" containerID="3d3452aa7ed632c71b41a89cdcb30a684c3577c11e08ecedad9676689947660e" Oct 30 07:45:05.887422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.887115 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerStarted","Data":"81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa"} Oct 30 07:45:05.889288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.889263 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" event={"ID":"085812ea-9161-4900-8bde-4c5c585d513f","Type":"ContainerStarted","Data":"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47"} Oct 30 07:45:05.889400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.889314 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:05.889400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.889342 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" event={"ID":"085812ea-9161-4900-8bde-4c5c585d513f","Type":"ContainerStarted","Data":"7065acfec8954fe7e75b5f0b5af91bdc1154a857b1eb2a339ee3219f3e8b7c20"} Oct 30 07:45:05.891144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.891113 5894 generic.go:356] "Generic (PLEG): container finished" podID="22679da9-0561-4760-a07b-6adff3f83ef0" containerID="8f6daec40dec54387635e4b023e647265aa6a0ef63e5ef3f88e4ef410d3f61c1" exitCode=60 Oct 30 07:45:05.891144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.891136 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerDied","Data":"8f6daec40dec54387635e4b023e647265aa6a0ef63e5ef3f88e4ef410d3f61c1"} Oct 30 07:45:05.891440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.891420 5894 scope.go:122] "RemoveContainer" containerID="8f6daec40dec54387635e4b023e647265aa6a0ef63e5ef3f88e4ef410d3f61c1" Oct 30 07:45:05.892913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.892892 5894 generic.go:356] "Generic (PLEG): container finished" podID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerID="a290f891af0510009473afbdf4ed0a205ccee5037a59130ef629569d2fdd8a80" exitCode=60 Oct 30 07:45:05.893026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.892940 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerDied","Data":"a290f891af0510009473afbdf4ed0a205ccee5037a59130ef629569d2fdd8a80"} Oct 30 07:45:05.893233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.893218 5894 scope.go:122] "RemoveContainer" containerID="a290f891af0510009473afbdf4ed0a205ccee5037a59130ef629569d2fdd8a80" Oct 30 07:45:05.897521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.897442 5894 generic.go:356] "Generic (PLEG): container finished" podID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerID="141a33db9fb514e2dd53ad1d8d564bd7ad33b0220b004da42f68bd181600ede6" exitCode=60 Oct 30 07:45:05.897521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.897509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerDied","Data":"141a33db9fb514e2dd53ad1d8d564bd7ad33b0220b004da42f68bd181600ede6"} Oct 30 07:45:05.897819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.897800 5894 scope.go:122] "RemoveContainer" containerID="141a33db9fb514e2dd53ad1d8d564bd7ad33b0220b004da42f68bd181600ede6" Oct 30 07:45:05.900088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.900061 5894 generic.go:356] "Generic (PLEG): container finished" podID="6e4cceaa-bc57-4862-8212-f6920f930785" containerID="bc1544db753102fa7fbb55cbdbc5609b4489b20706a3de5940a8b0a352c4db43" exitCode=60 Oct 30 07:45:05.900195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.900093 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" event={"ID":"6e4cceaa-bc57-4862-8212-f6920f930785","Type":"ContainerDied","Data":"bc1544db753102fa7fbb55cbdbc5609b4489b20706a3de5940a8b0a352c4db43"} Oct 30 07:45:05.900394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.900379 5894 scope.go:122] "RemoveContainer" containerID="bc1544db753102fa7fbb55cbdbc5609b4489b20706a3de5940a8b0a352c4db43" Oct 30 07:45:05.902563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.902533 5894 generic.go:356] "Generic (PLEG): container finished" podID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerID="ba8643d919e4d8185629f13e84ac585db88f69f4b52beb0ed3143a2bdd3e4d22" exitCode=60 Oct 30 07:45:05.902986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.902617 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerDied","Data":"ba8643d919e4d8185629f13e84ac585db88f69f4b52beb0ed3143a2bdd3e4d22"} Oct 30 07:45:05.902986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.902685 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerStarted","Data":"7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893"} Oct 30 07:45:05.904775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.904748 5894 generic.go:356] "Generic (PLEG): container finished" podID="5125cf24-506e-47c7-85f9-ff51fa746129" containerID="fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed" exitCode=60 Oct 30 07:45:05.904881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.904832 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerDied","Data":"fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed"} Oct 30 07:45:05.905045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.905030 5894 scope.go:122] "RemoveContainer" containerID="fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed" Oct 30 07:45:05.905356 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:05.905237 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4_odh-ai-gateway-infra(5125cf24-506e-47c7-85f9-ff51fa746129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" Oct 30 07:45:05.906564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.906535 5894 generic.go:356] "Generic (PLEG): container finished" podID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerID="afbc00b5c5528df67e2cf3d4bc33d7d258b997a4e8ac534398afb8818627008d" exitCode=60 Oct 30 07:45:05.906716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.906625 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" event={"ID":"024f2407-22e9-4298-ae1c-5c604dc4dd35","Type":"ContainerDied","Data":"afbc00b5c5528df67e2cf3d4bc33d7d258b997a4e8ac534398afb8818627008d"} Oct 30 07:45:05.906863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.906846 5894 scope.go:122] "RemoveContainer" containerID="afbc00b5c5528df67e2cf3d4bc33d7d258b997a4e8ac534398afb8818627008d" Oct 30 07:45:05.908566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.908548 5894 generic.go:356] "Generic (PLEG): container finished" podID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerID="817135c67be0dd642e137e85babc2bd275a96f962c4ca30ca34f4bb6e2532bd9" exitCode=60 Oct 30 07:45:05.908683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.908614 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerDied","Data":"817135c67be0dd642e137e85babc2bd275a96f962c4ca30ca34f4bb6e2532bd9"} Oct 30 07:45:05.909321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.908996 5894 scope.go:122] "RemoveContainer" containerID="817135c67be0dd642e137e85babc2bd275a96f962c4ca30ca34f4bb6e2532bd9" Oct 30 07:45:05.910487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.910464 5894 generic.go:356] "Generic (PLEG): container finished" podID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerID="d25ae25a292d577728b39047574ae17e126f28b5c4c61e700e79c446ec2dd71c" exitCode=60 Oct 30 07:45:05.910677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.910501 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerDied","Data":"d25ae25a292d577728b39047574ae17e126f28b5c4c61e700e79c446ec2dd71c"} Oct 30 07:45:05.910842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.910821 5894 scope.go:122] "RemoveContainer" containerID="d25ae25a292d577728b39047574ae17e126f28b5c4c61e700e79c446ec2dd71c" Oct 30 07:45:05.941757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:05.941733 5894 scope.go:122] "RemoveContainer" containerID="ffab1d50c965c98e8a83a00a517d1f32603e13f1a3951c97e4c30fdf2b97f4c2" Oct 30 07:45:05.968883 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:05.967392 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0f76b0/agc-cred-b42b42106144: secret "agc-cred-b42b42106144" not found Oct 30 07:45:05.968883 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:05.967419 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667: secret "agc-cred-b42b42106144" not found Oct 30 07:45:05.968883 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:05.967479 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:06.467457846 +0000 UTC m=+2870.158375275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : secret "agc-cred-b42b42106144" not found Oct 30 07:45:06.046184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.046137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:06.337970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.337914 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" podStartSLOduration=6.337898777 podStartE2EDuration="6.337898777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:06.336738264 +0000 UTC m=+2870.027655700" watchObservedRunningTime="2026-10-30 07:45:06.337898777 +0000 UTC m=+2870.028816281" Oct 30 07:45:06.424780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.424709 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" podStartSLOduration=3.424692485 podStartE2EDuration="3.424692485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:06.41966517 +0000 UTC m=+2870.110582608" watchObservedRunningTime="2026-10-30 07:45:06.424692485 +0000 UTC m=+2870.115609925" Oct 30 07:45:06.478419 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.478388 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0f76b0/agc-cred-b42b42106144: secret "agc-cred-b42b42106144" not found Oct 30 07:45:06.478419 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.478417 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667: secret "agc-cred-b42b42106144" not found Oct 30 07:45:06.478840 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.478474 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:07.478456328 +0000 UTC m=+2871.169373763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : secret "agc-cred-b42b42106144" not found Oct 30 07:45:06.751300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.751222 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:06.835527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.835490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:06.923682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.923617 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerStarted","Data":"9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a"} Oct 30 07:45:06.926317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.926286 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerStarted","Data":"2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d"} Oct 30 07:45:06.928723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.928687 5894 generic.go:356] "Generic (PLEG): container finished" podID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerID="81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa" exitCode=60 Oct 30 07:45:06.928870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.928754 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerDied","Data":"81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa"} Oct 30 07:45:06.928870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.928806 5894 scope.go:122] "RemoveContainer" containerID="699e2827890476df5b45480b46630bab177a2da0443acf95913d885d9c614952" Oct 30 07:45:06.928974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.928952 5894 scope.go:122] "RemoveContainer" containerID="81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa" Oct 30 07:45:06.929328 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.929301 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd_odh-ai-gateway-infra(89601e6b-88f3-4a7a-9d88-cc02c7c9b67c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" Oct 30 07:45:06.931086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.931062 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerStarted","Data":"35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775"} Oct 30 07:45:06.933535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.933496 5894 generic.go:356] "Generic (PLEG): container finished" podID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerID="7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893" exitCode=60 Oct 30 07:45:06.933671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.933606 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerDied","Data":"7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893"} Oct 30 07:45:06.934131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.934109 5894 scope.go:122] "RemoveContainer" containerID="7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893" Oct 30 07:45:06.934473 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.934439 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889105-r58rq_odh-ai-gateway-infra(cad6535c-446f-4f3c-b5b3-99262e02d40b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" Oct 30 07:45:06.936076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.936054 5894 scope.go:122] "RemoveContainer" containerID="fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed" Oct 30 07:45:06.936419 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.936394 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4_odh-ai-gateway-infra(5125cf24-506e-47c7-85f9-ff51fa746129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" Oct 30 07:45:06.938194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.938171 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerStarted","Data":"ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98"} Oct 30 07:45:06.940500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.940466 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerStarted","Data":"cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059"} Oct 30 07:45:06.942831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.942805 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerStarted","Data":"f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09"} Oct 30 07:45:06.946064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.946040 5894 generic.go:356] "Generic (PLEG): container finished" podID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerID="2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50" exitCode=60 Oct 30 07:45:06.946168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.946114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerDied","Data":"2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50"} Oct 30 07:45:06.946400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.946381 5894 scope.go:122] "RemoveContainer" containerID="2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50" Oct 30 07:45:06.946762 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.946721 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h_odh-ai-gateway-infra(4980eabc-1453-40d4-92b3-b3e8f489e1a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" Oct 30 07:45:06.947824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.947807 5894 scope.go:122] "RemoveContainer" containerID="ba8643d919e4d8185629f13e84ac585db88f69f4b52beb0ed3143a2bdd3e4d22" Oct 30 07:45:06.948266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.948250 5894 scope.go:122] "RemoveContainer" containerID="a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca" Oct 30 07:45:06.948358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.948341 5894 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" secret="" err="secret \"payload-processing-external-model-e2e--b7f37f47-dockercfg-q6p22\" not found" Oct 30 07:45:06.948553 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.948527 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj_odh-ai-gateway-infra(a1666912-5749-4286-bb70-75f9968f3cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" podUID="a1666912-5749-4286-bb70-75f9968f3cca" Oct 30 07:45:06.964757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:06.964728 5894 scope.go:122] "RemoveContainer" containerID="99a7a32d8a9dd6494bb18c37313391e11088000c94f91db39c3edec22b6d5827" Oct 30 07:45:06.987225 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.987191 5894 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-plugins-e2e-extmod-b-0f76b0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:06.987393 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:06.987279 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:07.487260388 +0000 UTC m=+2871.178177822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:07.226708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.226481 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" podStartSLOduration=7.226459752 podStartE2EDuration="7.226459752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:07.220158243 +0000 UTC m=+2870.911075680" watchObservedRunningTime="2026-10-30 07:45:07.226459752 +0000 UTC m=+2870.917377191" Oct 30 07:45:07.242967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.242933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:07.540782 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.540748 5894 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-plugins-e2e-extmod-b-0f76b0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:07.541252 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.540824 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:08.540806208 +0000 UTC m=+2872.231723624 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:07.541252 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.540754 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0f76b0/agc-cred-b42b42106144: secret "agc-cred-b42b42106144" not found Oct 30 07:45:07.541252 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.540867 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667: secret "agc-cred-b42b42106144" not found Oct 30 07:45:07.541252 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.540923 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:09.540906799 +0000 UTC m=+2873.231824228 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : secret "agc-cred-b42b42106144" not found Oct 30 07:45:07.782247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.782202 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:07.859253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.859212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:07.958833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.958800 5894 generic.go:356] "Generic (PLEG): container finished" podID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerID="6f6875eea45c428064e7066be457092b03b884d34f779cd8fa7a5690de5442cc" exitCode=60 Oct 30 07:45:07.959029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.958872 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerDied","Data":"6f6875eea45c428064e7066be457092b03b884d34f779cd8fa7a5690de5442cc"} Oct 30 07:45:07.959029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.958971 5894 scope.go:122] "RemoveContainer" containerID="817135c67be0dd642e137e85babc2bd275a96f962c4ca30ca34f4bb6e2532bd9" Oct 30 07:45:07.959414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.959393 5894 scope.go:122] "RemoveContainer" containerID="6f6875eea45c428064e7066be457092b03b884d34f779cd8fa7a5690de5442cc" Oct 30 07:45:07.959868 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.959839 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d_odh-ai-gateway-infra(38827e8c-bde0-4815-a8a0-a5b60c93a64f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" Oct 30 07:45:07.961334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.961313 5894 generic.go:356] "Generic (PLEG): container finished" podID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerID="ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98" exitCode=60 Oct 30 07:45:07.961463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.961406 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerDied","Data":"ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98"} Oct 30 07:45:07.961738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.961715 5894 scope.go:122] "RemoveContainer" containerID="ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98" Oct 30 07:45:07.962146 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.962117 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5_odh-ai-gateway-infra(bdd566e3-dde0-400a-a58a-5c6e37ad55ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" Oct 30 07:45:07.963701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.963675 5894 generic.go:356] "Generic (PLEG): container finished" podID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerID="cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059" exitCode=60 Oct 30 07:45:07.963812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.963698 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerDied","Data":"cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059"} Oct 30 07:45:07.964078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.964055 5894 scope.go:122] "RemoveContainer" containerID="cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059" Oct 30 07:45:07.964440 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.964414 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc_odh-ai-gateway-infra(fbf4fb4b-e362-4cae-8609-5557251e494d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" Oct 30 07:45:07.965932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.965914 5894 generic.go:356] "Generic (PLEG): container finished" podID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerID="f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09" exitCode=60 Oct 30 07:45:07.966024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.965981 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerDied","Data":"f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09"} Oct 30 07:45:07.966340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.966324 5894 scope.go:122] "RemoveContainer" containerID="f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09" Oct 30 07:45:07.966734 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.966707 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq_odh-ai-gateway-infra(d4667d03-9bf5-48ab-b817-23f8c241c7bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" Oct 30 07:45:07.968208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.968190 5894 scope.go:122] "RemoveContainer" containerID="2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50" Oct 30 07:45:07.968503 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.968484 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h_odh-ai-gateway-infra(4980eabc-1453-40d4-92b3-b3e8f489e1a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" Oct 30 07:45:07.969929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.969907 5894 generic.go:356] "Generic (PLEG): container finished" podID="1de640e8-db88-438e-8279-e7efd370deb6" containerID="9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a" exitCode=60 Oct 30 07:45:07.970036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.969980 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerDied","Data":"9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a"} Oct 30 07:45:07.970316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.970296 5894 scope.go:122] "RemoveContainer" containerID="9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a" Oct 30 07:45:07.970626 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.970590 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv_odh-ai-gateway-infra(1de640e8-db88-438e-8279-e7efd370deb6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" podUID="1de640e8-db88-438e-8279-e7efd370deb6" Oct 30 07:45:07.972571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.972542 5894 generic.go:356] "Generic (PLEG): container finished" podID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerID="2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d" exitCode=60 Oct 30 07:45:07.972709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.972617 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerDied","Data":"2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d"} Oct 30 07:45:07.972964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.972948 5894 scope.go:122] "RemoveContainer" containerID="2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d" Oct 30 07:45:07.973250 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.973224 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj_odh-ai-gateway-infra(8f6d77cd-1723-4b98-a138-d3cfc8de9b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" Oct 30 07:45:07.974788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.974767 5894 generic.go:356] "Generic (PLEG): container finished" podID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerID="15f159d6caa4ee65f754f21a36019348285f9897d3e5b8238a4aeddf39b5eadf" exitCode=60 Oct 30 07:45:07.974902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.974835 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerDied","Data":"15f159d6caa4ee65f754f21a36019348285f9897d3e5b8238a4aeddf39b5eadf"} Oct 30 07:45:07.975170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.975151 5894 scope.go:122] "RemoveContainer" containerID="15f159d6caa4ee65f754f21a36019348285f9897d3e5b8238a4aeddf39b5eadf" Oct 30 07:45:07.975413 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.975394 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv_odh-ai-gateway-infra(c4614d8a-8a8d-49d7-a91b-2c986228b6a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" Oct 30 07:45:07.976832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.976813 5894 generic.go:356] "Generic (PLEG): container finished" podID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerID="d73ede091f00739e954576997e4cadc1132ab861433fad95cd39447f62c271a7" exitCode=60 Oct 30 07:45:07.976906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.976874 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerDied","Data":"d73ede091f00739e954576997e4cadc1132ab861433fad95cd39447f62c271a7"} Oct 30 07:45:07.977186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.977173 5894 scope.go:122] "RemoveContainer" containerID="d73ede091f00739e954576997e4cadc1132ab861433fad95cd39447f62c271a7" Oct 30 07:45:07.977433 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.977411 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr_odh-ai-gateway-infra(fde8b11e-629f-488c-82a3-90ab92d49c4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" Oct 30 07:45:07.980296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.980279 5894 scope.go:122] "RemoveContainer" containerID="81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa" Oct 30 07:45:07.980519 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.980500 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd_odh-ai-gateway-infra(89601e6b-88f3-4a7a-9d88-cc02c7c9b67c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" Oct 30 07:45:07.982225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.982206 5894 generic.go:356] "Generic (PLEG): container finished" podID="22679da9-0561-4760-a07b-6adff3f83ef0" containerID="0a862868ad688f07999a3fb110825da9498db3d6f21fc52b2f2fa422ef597e34" exitCode=60 Oct 30 07:45:07.982321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.982264 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerDied","Data":"0a862868ad688f07999a3fb110825da9498db3d6f21fc52b2f2fa422ef597e34"} Oct 30 07:45:07.982566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.982546 5894 scope.go:122] "RemoveContainer" containerID="0a862868ad688f07999a3fb110825da9498db3d6f21fc52b2f2fa422ef597e34" Oct 30 07:45:07.982863 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.982839 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d_odh-ai-gateway-infra(22679da9-0561-4760-a07b-6adff3f83ef0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" Oct 30 07:45:07.984464 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.984441 5894 generic.go:356] "Generic (PLEG): container finished" podID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerID="35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775" exitCode=60 Oct 30 07:45:07.984557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.984510 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerDied","Data":"35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775"} Oct 30 07:45:07.984856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.984838 5894 scope.go:122] "RemoveContainer" containerID="35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775" Oct 30 07:45:07.985130 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.985110 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd_odh-ai-gateway-infra(c895d4bf-e5f2-44cb-959f-d36c5b14e623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" Oct 30 07:45:07.986456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.986433 5894 generic.go:356] "Generic (PLEG): container finished" podID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerID="c7f7aaa70bd784695940fef485d22cc2e3054fe2377ca8a7b7056f99765c9610" exitCode=60 Oct 30 07:45:07.986554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.986508 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerDied","Data":"c7f7aaa70bd784695940fef485d22cc2e3054fe2377ca8a7b7056f99765c9610"} Oct 30 07:45:07.986847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.986829 5894 scope.go:122] "RemoveContainer" containerID="c7f7aaa70bd784695940fef485d22cc2e3054fe2377ca8a7b7056f99765c9610" Oct 30 07:45:07.987083 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.987062 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4_odh-ai-gateway-infra(a4eced12-8c18-4a34-9c19-ea57979730cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" Oct 30 07:45:07.988894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.988867 5894 generic.go:356] "Generic (PLEG): container finished" podID="6e4cceaa-bc57-4862-8212-f6920f930785" containerID="2282bc0a53030853bbcb20cc25ad0819fa251a29ecce1bd8b71aba9c5a73760d" exitCode=60 Oct 30 07:45:07.988982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.988910 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" event={"ID":"6e4cceaa-bc57-4862-8212-f6920f930785","Type":"ContainerDied","Data":"2282bc0a53030853bbcb20cc25ad0819fa251a29ecce1bd8b71aba9c5a73760d"} Oct 30 07:45:07.989209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.989195 5894 scope.go:122] "RemoveContainer" containerID="2282bc0a53030853bbcb20cc25ad0819fa251a29ecce1bd8b71aba9c5a73760d" Oct 30 07:45:07.989495 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.989475 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9_odh-ai-gateway-infra(6e4cceaa-bc57-4862-8212-f6920f930785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" Oct 30 07:45:07.992821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.992797 5894 generic.go:356] "Generic (PLEG): container finished" podID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerID="fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e" exitCode=60 Oct 30 07:45:07.992924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.992877 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" event={"ID":"024f2407-22e9-4298-ae1c-5c604dc4dd35","Type":"ContainerDied","Data":"fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e"} Oct 30 07:45:07.993052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:07.993037 5894 scope.go:122] "RemoveContainer" containerID="fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e" Oct 30 07:45:07.993257 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:07.993241 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg_odh-ai-gateway-infra(024f2407-22e9-4298-ae1c-5c604dc4dd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" Oct 30 07:45:08.232414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.229849 5894 scope.go:122] "RemoveContainer" containerID="d25ae25a292d577728b39047574ae17e126f28b5c4c61e700e79c446ec2dd71c" Oct 30 07:45:08.319240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.319212 5894 scope.go:122] "RemoveContainer" containerID="b16d49cc9835f82121170151fabb8fe44041ed4e3e9345c17028021232af7410" Oct 30 07:45:08.341143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.340258 5894 scope.go:122] "RemoveContainer" containerID="4160472349a6fce042d1879a17baa4e0e031ffb85b817dbfc465f9d8332461d7" Oct 30 07:45:08.357697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.357660 5894 scope.go:122] "RemoveContainer" containerID="70efa65f3b576616ed25082660453fb1915111a7d910805fb3a1d21292d7759c" Oct 30 07:45:08.377300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.377275 5894 scope.go:122] "RemoveContainer" containerID="1ce9d064559f8afb2a457ad2c0a6d4b7c973eff1869798f5e6776a2e25ed69a6" Oct 30 07:45:08.396432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.396410 5894 scope.go:122] "RemoveContainer" containerID="23268e8c1bc72c7af8f2ef5b66c3083c93e8b2bbde8b5987f29b64c0590cb87e" Oct 30 07:45:08.414269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.414211 5894 scope.go:122] "RemoveContainer" containerID="3d3452aa7ed632c71b41a89cdcb30a684c3577c11e08ecedad9676689947660e" Oct 30 07:45:08.431979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.431957 5894 scope.go:122] "RemoveContainer" containerID="8f6daec40dec54387635e4b023e647265aa6a0ef63e5ef3f88e4ef410d3f61c1" Oct 30 07:45:08.448042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.447845 5894 scope.go:122] "RemoveContainer" containerID="a290f891af0510009473afbdf4ed0a205ccee5037a59130ef629569d2fdd8a80" Oct 30 07:45:08.463549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.463526 5894 scope.go:122] "RemoveContainer" containerID="141a33db9fb514e2dd53ad1d8d564bd7ad33b0220b004da42f68bd181600ede6" Oct 30 07:45:08.478936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.478912 5894 scope.go:122] "RemoveContainer" containerID="bc1544db753102fa7fbb55cbdbc5609b4489b20706a3de5940a8b0a352c4db43" Oct 30 07:45:08.497124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:08.497099 5894 scope.go:122] "RemoveContainer" containerID="afbc00b5c5528df67e2cf3d4bc33d7d258b997a4e8ac534398afb8818627008d" Oct 30 07:45:08.568008 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:08.567971 5894 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-plugins-e2e-extmod-b-0f76b0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:08.568414 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:08.568051 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:10.568035767 +0000 UTC m=+2874.258953182 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:09.018499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:09.018468 5894 scope.go:122] "RemoveContainer" containerID="fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e" Oct 30 07:45:09.019016 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.018971 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg_odh-ai-gateway-infra(024f2407-22e9-4298-ae1c-5c604dc4dd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" Oct 30 07:45:09.020232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:09.020195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:09.021843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:09.021820 5894 scope.go:122] "RemoveContainer" containerID="ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98" Oct 30 07:45:09.022258 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.022229 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5_odh-ai-gateway-infra(bdd566e3-dde0-400a-a58a-5c6e37ad55ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" Oct 30 07:45:09.023398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:09.023379 5894 scope.go:122] "RemoveContainer" containerID="cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059" Oct 30 07:45:09.023765 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.023742 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc_odh-ai-gateway-infra(fbf4fb4b-e362-4cae-8609-5557251e494d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" Oct 30 07:45:09.585874 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.585602 5894 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-0f76b0/agc-cred-b42b42106144: secret "agc-cred-b42b42106144" not found Oct 30 07:45:09.585874 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.585875 5894 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667: secret "agc-cred-b42b42106144" not found Oct 30 07:45:09.586385 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:09.585957 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:13.585936041 +0000 UTC m=+2877.276853475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : secret "agc-cred-b42b42106144" not found Oct 30 07:45:10.602196 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:10.602157 5894 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-plugins-e2e-extmod-b-0f76b0: configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:10.602615 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:10.602240 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume podName:085812ea-9161-4900-8bde-4c5c585d513f nodeName:}" failed. No retries permitted until 2026-10-30 07:45:14.602223472 +0000 UTC m=+2878.293140887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" (UID: "085812ea-9161-4900-8bde-4c5c585d513f") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-0f76b0" not found Oct 30 07:45:10.638372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:10.638336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:12.260107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:12.260074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:12.383185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:12.383148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:12.782874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:12.782827 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" podUID="1b3448b7-cb81-416d-9793-e4e719e638a1" containerName="payload-processing" containerID="cri-o://e3bfde9be2bc95d506c1131f1b84466b942f45a73b127b463e77a0ddc62e186e" gracePeriod=25 Oct 30 07:45:12.802714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:12.802681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:12.993510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:12.993388 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" podUID="085812ea-9161-4900-8bde-4c5c585d513f" containerName="payload-processing" containerID="cri-o://8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47" gracePeriod=25 Oct 30 07:45:13.065013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.064982 5894 generic.go:356] "Generic (PLEG): container finished" podID="1b3448b7-cb81-416d-9793-e4e719e638a1" containerID="e3bfde9be2bc95d506c1131f1b84466b942f45a73b127b463e77a0ddc62e186e" exitCode=0 Oct 30 07:45:13.065202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.065070 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" event={"ID":"1b3448b7-cb81-416d-9793-e4e719e638a1","Type":"ContainerDied","Data":"e3bfde9be2bc95d506c1131f1b84466b942f45a73b127b463e77a0ddc62e186e"} Oct 30 07:45:13.139369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.139333 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:13.333150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.268921 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\") pod \"1b3448b7-cb81-416d-9793-e4e719e638a1\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " Oct 30 07:45:13.333150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.269414 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\") pod \"1b3448b7-cb81-416d-9793-e4e719e638a1\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " Oct 30 07:45:13.333150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.269571 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\") pod \"1b3448b7-cb81-416d-9793-e4e719e638a1\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " Oct 30 07:45:13.333150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.269664 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\") pod \"1b3448b7-cb81-416d-9793-e4e719e638a1\" (UID: \"1b3448b7-cb81-416d-9793-e4e719e638a1\") " Oct 30 07:45:13.334081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.334022 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume" pod "1b3448b7-cb81-416d-9793-e4e719e638a1" (UID: "1b3448b7-cb81-416d-9793-e4e719e638a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:13.354552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.354494 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume" pod "1b3448b7-cb81-416d-9793-e4e719e638a1" (UID: "1b3448b7-cb81-416d-9793-e4e719e638a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:13.373072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.373019 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.444669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.436704 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1b3448b7-cb81-416d-9793-e4e719e638a1-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.444992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.429001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:13.445186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.384943 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg" pod "1b3448b7-cb81-416d-9793-e4e719e638a1" (UID: "1b3448b7-cb81-416d-9793-e4e719e638a1"). InnerVolumeSpecName "kube-api-access-bvfrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:13.445320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.421820 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials" pod "1b3448b7-cb81-416d-9793-e4e719e638a1" (UID: "1b3448b7-cb81-416d-9793-e4e719e638a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:13.548472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.548440 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:13.549431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.549407 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfrg\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-kube-api-access-bvfrg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.549625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.549610 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1b3448b7-cb81-416d-9793-e4e719e638a1-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.650832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.650744 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\") pod \"085812ea-9161-4900-8bde-4c5c585d513f\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " Oct 30 07:45:13.651001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.650909 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\") pod \"085812ea-9161-4900-8bde-4c5c585d513f\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " Oct 30 07:45:13.651064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651017 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\") pod \"085812ea-9161-4900-8bde-4c5c585d513f\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " Oct 30 07:45:13.651168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651150 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\") pod \"085812ea-9161-4900-8bde-4c5c585d513f\" (UID: \"085812ea-9161-4900-8bde-4c5c585d513f\") " Oct 30 07:45:13.651237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651183 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume" pod "085812ea-9161-4900-8bde-4c5c585d513f" (UID: "085812ea-9161-4900-8bde-4c5c585d513f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:13.651294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651278 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume" pod "085812ea-9161-4900-8bde-4c5c585d513f" (UID: "085812ea-9161-4900-8bde-4c5c585d513f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:13.651904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651880 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.651904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.651904 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/085812ea-9161-4900-8bde-4c5c585d513f-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.653882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.653852 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials" pod "085812ea-9161-4900-8bde-4c5c585d513f" (UID: "085812ea-9161-4900-8bde-4c5c585d513f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:13.653977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.653891 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv" pod "085812ea-9161-4900-8bde-4c5c585d513f" (UID: "085812ea-9161-4900-8bde-4c5c585d513f"). InnerVolumeSpecName "kube-api-access-9sjfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:13.760366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.760328 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.760582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.760568 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjfv\" (UniqueName: \"kubernetes.io/projected/085812ea-9161-4900-8bde-4c5c585d513f-kube-api-access-9sjfv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:13.829556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:13.829516 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:14.077994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.077964 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" Oct 30 07:45:14.078189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.078003 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc" event={"ID":"1b3448b7-cb81-416d-9793-e4e719e638a1","Type":"ContainerDied","Data":"7caec6f82abec68af51ee99513dd9c9291604570311579563ba31f48ecde6078"} Oct 30 07:45:14.078189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.078073 5894 scope.go:122] "RemoveContainer" containerID="e3bfde9be2bc95d506c1131f1b84466b942f45a73b127b463e77a0ddc62e186e" Oct 30 07:45:14.085599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.080912 5894 generic.go:356] "Generic (PLEG): container finished" podID="085812ea-9161-4900-8bde-4c5c585d513f" containerID="8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47" exitCode=0 Oct 30 07:45:14.085599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.081149 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" event={"ID":"085812ea-9161-4900-8bde-4c5c585d513f","Type":"ContainerDied","Data":"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47"} Oct 30 07:45:14.085599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.081192 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" event={"ID":"085812ea-9161-4900-8bde-4c5c585d513f","Type":"ContainerDied","Data":"7065acfec8954fe7e75b5f0b5af91bdc1154a857b1eb2a339ee3219f3e8b7c20"} Oct 30 07:45:14.085599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.081257 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667" Oct 30 07:45:14.096374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.096339 5894 scope.go:122] "RemoveContainer" containerID="8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47" Oct 30 07:45:14.109287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.109266 5894 scope.go:122] "RemoveContainer" containerID="8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47" Oct 30 07:45:14.109666 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:14.109624 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47\": container with ID starting with 8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47 not found: ID does not exist" containerID="8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47" Oct 30 07:45:14.109738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.109678 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47"} err="failed to get container status \"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47\": rpc error: code = NotFound desc = could not find container \"8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47\": container with ID starting with 8a94a15e90e4967191c5ab712517d212646d67e0e6ccf457d64f48d7d8901b47 not found: ID does not exist" Oct 30 07:45:14.124213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.124182 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:14.136570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.136538 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:14.149764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.149737 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-b096ec/payload-processing-external-model-e2e-extmod-a-b096ec-6f4b9qsnc"] Oct 30 07:45:14.209656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.198809 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:14.209656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.203889 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-0f76b0/payload-processing-external-model-e2e-extmod-b-0f76b0-779cff667"] Oct 30 07:45:14.434394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.434313 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085812ea-9161-4900-8bde-4c5c585d513f" path="/var/lib/kubelet/pods/085812ea-9161-4900-8bde-4c5c585d513f/volumes" Oct 30 07:45:14.434853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:14.434836 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b3448b7-cb81-416d-9793-e4e719e638a1" path="/var/lib/kubelet/pods/1b3448b7-cb81-416d-9793-e4e719e638a1/volumes" Oct 30 07:45:15.963052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:15.962942 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:16.436466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:16.436435 5894 scope.go:122] "RemoveContainer" containerID="a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca" Oct 30 07:45:17.113529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:17.113441 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerStarted","Data":"36dc542d03eeb1d7702f16b26b036d7f7c507d201fa4696367909a437d0b191b"} Oct 30 07:45:17.303755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:17.303713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:17.478611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:17.478515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:17.894447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:17.894411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:18.124034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.124002 5894 generic.go:356] "Generic (PLEG): container finished" podID="a1666912-5749-4286-bb70-75f9968f3cca" containerID="36dc542d03eeb1d7702f16b26b036d7f7c507d201fa4696367909a437d0b191b" exitCode=60 Oct 30 07:45:18.124449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.124074 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerDied","Data":"36dc542d03eeb1d7702f16b26b036d7f7c507d201fa4696367909a437d0b191b"} Oct 30 07:45:18.124449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.124134 5894 scope.go:122] "RemoveContainer" containerID="a9c6381aa1a802cb6a61d3cf19880e74b14f17d3ecc0887e80a1816fa0cd93ca" Oct 30 07:45:18.124591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.124575 5894 scope.go:122] "RemoveContainer" containerID="36dc542d03eeb1d7702f16b26b036d7f7c507d201fa4696367909a437d0b191b" Oct 30 07:45:18.125037 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:18.125013 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj_odh-ai-gateway-infra(a1666912-5749-4286-bb70-75f9968f3cca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" podUID="a1666912-5749-4286-bb70-75f9968f3cca" Oct 30 07:45:18.179045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.178951 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:18.435609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.435513 5894 scope.go:122] "RemoveContainer" containerID="6f6875eea45c428064e7066be457092b03b884d34f779cd8fa7a5690de5442cc" Oct 30 07:45:18.435833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.435743 5894 scope.go:122] "RemoveContainer" containerID="f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09" Oct 30 07:45:18.435951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.435927 5894 scope.go:122] "RemoveContainer" containerID="7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893" Oct 30 07:45:18.436089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.436071 5894 scope.go:122] "RemoveContainer" containerID="9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a" Oct 30 07:45:18.436216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.436199 5894 scope.go:122] "RemoveContainer" containerID="d73ede091f00739e954576997e4cadc1132ab861433fad95cd39447f62c271a7" Oct 30 07:45:18.436374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:18.436299 5894 scope.go:122] "RemoveContainer" containerID="cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059" Oct 30 07:45:19.073196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.072258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:19.137081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.137042 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerStarted","Data":"4dc3e40c7f3075c14e4f023bec1653a51537a48ba462165e9e5db5f74dace4f9"} Oct 30 07:45:19.143847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.143807 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerStarted","Data":"c574b6ea4567be58feb8dc5cbe5694e0b9b5afbce4c1733d4a5efd77c08fcf83"} Oct 30 07:45:19.153694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.146390 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerStarted","Data":"3e31f3faa8e8ceddbf0c69bf73dd6c10ae0143227fd7fac8af1a6aefc4a0d06d"} Oct 30 07:45:19.153694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.148919 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerStarted","Data":"afb473c64f4e1632aa3edcbee304bb17758f0756cd3deb22e4d85be852275406"} Oct 30 07:45:19.153694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.151414 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerStarted","Data":"525b54904e34912d8f6524da9e5c428a517a09d2f1456a75fcd65c995a1c071f"} Oct 30 07:45:19.153905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.153706 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerStarted","Data":"e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520"} Oct 30 07:45:19.194745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.194681 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:19.262766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.262690 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:45:19.263703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263676 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1b3448b7-cb81-416d-9793-e4e719e638a1" containerName="payload-processing" Oct 30 07:45:19.263703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263701 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b3448b7-cb81-416d-9793-e4e719e638a1" containerName="payload-processing" Oct 30 07:45:19.263904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263784 5894 cpu_manager.go:414] "containerMap: removing container" podUID="085812ea-9161-4900-8bde-4c5c585d513f" containerName="payload-processing" Oct 30 07:45:19.263904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263790 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085812ea-9161-4900-8bde-4c5c585d513f" containerName="payload-processing" Oct 30 07:45:19.264013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263948 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="085812ea-9161-4900-8bde-4c5c585d513f" containerName="payload-processing" Oct 30 07:45:19.264013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.263960 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b3448b7-cb81-416d-9793-e4e719e638a1" containerName="payload-processing" Oct 30 07:45:19.273785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.273743 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.292176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.289868 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fc3a23-dockercfg-n8wpt\"" Oct 30 07:45:19.301003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.294597 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fc3a23\"" Oct 30 07:45:19.371845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.344034 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.508583 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.508904 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm7wm\" (UniqueName: \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.509402 5894 scope.go:122] "RemoveContainer" containerID="2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.509526 5894 scope.go:122] "RemoveContainer" containerID="ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.509613 5894 scope.go:122] "RemoveContainer" containerID="2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.509731 5894 scope.go:122] "RemoveContainer" containerID="35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775" Oct 30 07:45:19.520724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.509884 5894 scope.go:122] "RemoveContainer" containerID="0a862868ad688f07999a3fb110825da9498db3d6f21fc52b2f2fa422ef597e34" Oct 30 07:45:19.592419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.590773 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:45:19.613866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.613721 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.614157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.614131 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm7wm\" (UniqueName: \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.615683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.615658 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.618044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.617923 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.618155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.617842 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:45:19.646281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.646220 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm7wm\" (UniqueName: \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\") pod \"payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.648093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.648066 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:19.676618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.676591 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:19.696644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.696505 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:19.712604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.712572 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:19.716654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.716558 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.717135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.717110 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrk9s\" (UniqueName: \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.818280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.818222 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrk9s\" (UniqueName: \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.818563 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.818514 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.819869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.819836 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.845316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.845250 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrk9s\" (UniqueName: \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\") pod \"payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:19.846350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.846327 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:19.847210 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:19.847186 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf4fb4b_e362_4cae_8609_5557251e494d.slice/crio-e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:45:19.868135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:19.868095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:20.032689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.031769 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:20.042198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.042169 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\") pod \"a1666912-5749-4286-bb70-75f9968f3cca\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " Oct 30 07:45:20.042396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.042377 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\" (UniqueName: \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\") pod \"a1666912-5749-4286-bb70-75f9968f3cca\" (UID: \"a1666912-5749-4286-bb70-75f9968f3cca\") " Oct 30 07:45:20.043522 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.043482 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca" pod "a1666912-5749-4286-bb70-75f9968f3cca" (UID: "a1666912-5749-4286-bb70-75f9968f3cca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.047598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.047554 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7" pod "a1666912-5749-4286-bb70-75f9968f3cca" (UID: "a1666912-5749-4286-bb70-75f9968f3cca"). InnerVolumeSpecName "kube-api-access-5cnq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.084232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.084185 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:20.144281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.144235 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnq7\" (UniqueName: \"kubernetes.io/projected/a1666912-5749-4286-bb70-75f9968f3cca-kube-api-access-5cnq7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.144281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.144272 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1666912-5749-4286-bb70-75f9968f3cca-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.169057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.169021 5894 generic.go:356] "Generic (PLEG): container finished" podID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerID="525b54904e34912d8f6524da9e5c428a517a09d2f1456a75fcd65c995a1c071f" exitCode=60 Oct 30 07:45:20.169260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.169110 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerDied","Data":"525b54904e34912d8f6524da9e5c428a517a09d2f1456a75fcd65c995a1c071f"} Oct 30 07:45:20.169260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.169191 5894 scope.go:122] "RemoveContainer" containerID="6f6875eea45c428064e7066be457092b03b884d34f779cd8fa7a5690de5442cc" Oct 30 07:45:20.170751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.169953 5894 scope.go:122] "RemoveContainer" containerID="525b54904e34912d8f6524da9e5c428a517a09d2f1456a75fcd65c995a1c071f" Oct 30 07:45:20.170751 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:20.170406 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d_odh-ai-gateway-infra(38827e8c-bde0-4815-a8a0-a5b60c93a64f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" Oct 30 07:45:20.173837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.172787 5894 generic.go:356] "Generic (PLEG): container finished" podID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerID="e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520" exitCode=60 Oct 30 07:45:20.173837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.172888 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerDied","Data":"e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520"} Oct 30 07:45:20.173837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.173355 5894 scope.go:122] "RemoveContainer" containerID="e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520" Oct 30 07:45:20.173837 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:20.173771 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc_odh-ai-gateway-infra(fbf4fb4b-e362-4cae-8609-5557251e494d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" Oct 30 07:45:20.176590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.176567 5894 generic.go:356] "Generic (PLEG): container finished" podID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerID="4dc3e40c7f3075c14e4f023bec1653a51537a48ba462165e9e5db5f74dace4f9" exitCode=60 Oct 30 07:45:20.176850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.176829 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerDied","Data":"4dc3e40c7f3075c14e4f023bec1653a51537a48ba462165e9e5db5f74dace4f9"} Oct 30 07:45:20.185566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.185543 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" Oct 30 07:45:20.185719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.185548 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj" event={"ID":"a1666912-5749-4286-bb70-75f9968f3cca","Type":"ContainerDied","Data":"7fee7612e49c008f8fa3ff2ee5fc91933451e973de5509d48aba86fb0484d181"} Oct 30 07:45:20.188686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.188515 5894 generic.go:356] "Generic (PLEG): container finished" podID="1de640e8-db88-438e-8279-e7efd370deb6" containerID="c574b6ea4567be58feb8dc5cbe5694e0b9b5afbce4c1733d4a5efd77c08fcf83" exitCode=60 Oct 30 07:45:20.188686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.188624 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerDied","Data":"c574b6ea4567be58feb8dc5cbe5694e0b9b5afbce4c1733d4a5efd77c08fcf83"} Oct 30 07:45:20.191824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.191301 5894 generic.go:356] "Generic (PLEG): container finished" podID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerID="3e31f3faa8e8ceddbf0c69bf73dd6c10ae0143227fd7fac8af1a6aefc4a0d06d" exitCode=60 Oct 30 07:45:20.191824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.191460 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerDied","Data":"3e31f3faa8e8ceddbf0c69bf73dd6c10ae0143227fd7fac8af1a6aefc4a0d06d"} Oct 30 07:45:20.200924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.200892 5894 generic.go:356] "Generic (PLEG): container finished" podID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerID="afb473c64f4e1632aa3edcbee304bb17758f0756cd3deb22e4d85be852275406" exitCode=60 Oct 30 07:45:20.201051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.201010 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerDied","Data":"afb473c64f4e1632aa3edcbee304bb17758f0756cd3deb22e4d85be852275406"} Oct 30 07:45:20.234003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.233899 5894 scope.go:122] "RemoveContainer" containerID="cfed1cf33145e4f742b320681e5862539fad62a0da448f01150fba5ef7fcc059" Oct 30 07:45:20.284278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.263991 5894 scope.go:122] "RemoveContainer" containerID="f7c0434551c5daa79b6c2458792bf475cefe3ab4d9c1d4525d28fcb272074e09" Oct 30 07:45:20.303819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.303753 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:45:20.306471 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:20.306438 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d5f6ca_4068_4c97_bbef_18b53675ff62.slice/crio-d1556cd895c579e50222962e4f53d2cd3c9d713cce559950c0bcfe0d8087e2d3 WatchSource:0}: Error finding container d1556cd895c579e50222962e4f53d2cd3c9d713cce559950c0bcfe0d8087e2d3: Status 404 returned error can't find the container with id d1556cd895c579e50222962e4f53d2cd3c9d713cce559950c0bcfe0d8087e2d3 Oct 30 07:45:20.332080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.330317 5894 scope.go:122] "RemoveContainer" containerID="36dc542d03eeb1d7702f16b26b036d7f7c507d201fa4696367909a437d0b191b" Oct 30 07:45:20.361862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.361829 5894 scope.go:122] "RemoveContainer" containerID="9f41c29e2ed001671a52816cda3d5a21383daeccd418f9084df24621b819d56a" Oct 30 07:45:20.416949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.416916 5894 scope.go:122] "RemoveContainer" containerID="d73ede091f00739e954576997e4cadc1132ab861433fad95cd39447f62c271a7" Oct 30 07:45:20.420818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.420450 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:20.424216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.424187 5894 scope.go:122] "RemoveContainer" containerID="2282bc0a53030853bbcb20cc25ad0819fa251a29ecce1bd8b71aba9c5a73760d" Oct 30 07:45:20.426526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.425211 5894 scope.go:122] "RemoveContainer" containerID="fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed" Oct 30 07:45:20.426526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.426074 5894 scope.go:122] "RemoveContainer" containerID="15f159d6caa4ee65f754f21a36019348285f9897d3e5b8238a4aeddf39b5eadf" Oct 30 07:45:20.426526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.426238 5894 scope.go:122] "RemoveContainer" containerID="81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa" Oct 30 07:45:20.427660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.427268 5894 scope.go:122] "RemoveContainer" containerID="fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e" Oct 30 07:45:20.514687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.514662 5894 scope.go:122] "RemoveContainer" containerID="7612926ec4522ed28f6061dab9a3a7620fd853424fc69986fc73061fec989893" Oct 30 07:45:20.533236 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:45:20.533197 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542c41c5_8a38_41c3_824f_025144acfc4f.slice/crio-4a2165deeaf4fdeaa5f302cc98ce595c582830598079966ead3ce5ba071455e7 WatchSource:0}: Error finding container 4a2165deeaf4fdeaa5f302cc98ce595c582830598079966ead3ce5ba071455e7: Status 404 returned error can't find the container with id 4a2165deeaf4fdeaa5f302cc98ce595c582830598079966ead3ce5ba071455e7 Oct 30 07:45:20.537143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.537114 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:45:20.537893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.537868 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.541954 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.544389 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.548306 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889105-895vj"] Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.552420 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\") pod \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.552500 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\" (UniqueName: \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\") pod \"1de640e8-db88-438e-8279-e7efd370deb6\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.552711 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\") pod \"1de640e8-db88-438e-8279-e7efd370deb6\" (UID: \"1de640e8-db88-438e-8279-e7efd370deb6\") " Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.552780 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\") pod \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\" (UID: \"d4667d03-9bf5-48ab-b817-23f8c241c7bf\") " Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.553664 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca" pod "1de640e8-db88-438e-8279-e7efd370deb6" (UID: "1de640e8-db88-438e-8279-e7efd370deb6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.553806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.553750 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca" pod "d4667d03-9bf5-48ab-b817-23f8c241c7bf" (UID: "d4667d03-9bf5-48ab-b817-23f8c241c7bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.561438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.556965 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf" pod "1de640e8-db88-438e-8279-e7efd370deb6" (UID: "1de640e8-db88-438e-8279-e7efd370deb6"). InnerVolumeSpecName "kube-api-access-nkqzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.561438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.558619 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67" pod "d4667d03-9bf5-48ab-b817-23f8c241c7bf" (UID: "d4667d03-9bf5-48ab-b817-23f8c241c7bf"). InnerVolumeSpecName "kube-api-access-rkq67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.561438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.559233 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:20.567561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.567536 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:20.598198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.598094 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:20.654230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.654205 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\" (UniqueName: \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\") pod \"cad6535c-446f-4f3c-b5b3-99262e02d40b\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " Oct 30 07:45:20.654430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.654418 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\") pod \"fde8b11e-629f-488c-82a3-90ab92d49c4f\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " Oct 30 07:45:20.654478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.654445 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\" (UniqueName: \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\") pod \"fde8b11e-629f-488c-82a3-90ab92d49c4f\" (UID: \"fde8b11e-629f-488c-82a3-90ab92d49c4f\") " Oct 30 07:45:20.654514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.654491 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\") pod \"cad6535c-446f-4f3c-b5b3-99262e02d40b\" (UID: \"cad6535c-446f-4f3c-b5b3-99262e02d40b\") " Oct 30 07:45:20.655277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.655045 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkq67\" (UniqueName: \"kubernetes.io/projected/d4667d03-9bf5-48ab-b817-23f8c241c7bf-kube-api-access-rkq67\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.655277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.655072 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkqzf\" (UniqueName: \"kubernetes.io/projected/1de640e8-db88-438e-8279-e7efd370deb6-kube-api-access-nkqzf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.655277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.655085 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1de640e8-db88-438e-8279-e7efd370deb6-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.655277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.655099 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4667d03-9bf5-48ab-b817-23f8c241c7bf-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.656152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.656012 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca" pod "cad6535c-446f-4f3c-b5b3-99262e02d40b" (UID: "cad6535c-446f-4f3c-b5b3-99262e02d40b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.658320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.658086 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr" pod "cad6535c-446f-4f3c-b5b3-99262e02d40b" (UID: "cad6535c-446f-4f3c-b5b3-99262e02d40b"). InnerVolumeSpecName "kube-api-access-fs6kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.659079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.658973 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc" pod "fde8b11e-629f-488c-82a3-90ab92d49c4f" (UID: "fde8b11e-629f-488c-82a3-90ab92d49c4f"). InnerVolumeSpecName "kube-api-access-p7ftc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:20.659860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.659838 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca" pod "fde8b11e-629f-488c-82a3-90ab92d49c4f" (UID: "fde8b11e-629f-488c-82a3-90ab92d49c4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:20.757814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.757721 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde8b11e-629f-488c-82a3-90ab92d49c4f-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.757814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.757756 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7ftc\" (UniqueName: \"kubernetes.io/projected/fde8b11e-629f-488c-82a3-90ab92d49c4f-kube-api-access-p7ftc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.757814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.757772 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cad6535c-446f-4f3c-b5b3-99262e02d40b-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:20.757814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:20.757787 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs6kr\" (UniqueName: \"kubernetes.io/projected/cad6535c-446f-4f3c-b5b3-99262e02d40b-kube-api-access-fs6kr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.213904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.213859 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" Oct 30 07:45:21.214382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.213869 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv" event={"ID":"1de640e8-db88-438e-8279-e7efd370deb6","Type":"ContainerDied","Data":"e382765d1fd3708c3d811327e444b3322dcc47a0ee30390948b1b98cea53a271"} Oct 30 07:45:21.214382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.214025 5894 scope.go:122] "RemoveContainer" containerID="c574b6ea4567be58feb8dc5cbe5694e0b9b5afbce4c1733d4a5efd77c08fcf83" Oct 30 07:45:21.217085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.217055 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerStarted","Data":"7801e1db0ced34265e2defd6255a51356eeaf4dcf014217c41a63de783c751e5"} Oct 30 07:45:21.220796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.220768 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerStarted","Data":"a73731e1bae4c40b0e44e70fd6b9515f2837c47f2cec3cd9f74179541aedf116"} Oct 30 07:45:21.222549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.222522 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" event={"ID":"fde8b11e-629f-488c-82a3-90ab92d49c4f","Type":"ContainerDied","Data":"cd152051d063f848c099088087f2c32d008a2d752a93c244539164f794136539"} Oct 30 07:45:21.222692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.222647 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr" Oct 30 07:45:21.233248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.233219 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerStarted","Data":"ceb95d73786675c0772836c164bd44470a4c12fa9175b741dd2c20c927d53e4a"} Oct 30 07:45:21.236582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.236555 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerStarted","Data":"3a0b3b63d4b780959e6214042ecf82bf6ee32aa99675547cd5860e6c68bdc7fd"} Oct 30 07:45:21.238595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.238564 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" event={"ID":"542c41c5-8a38-41c3-824f-025144acfc4f","Type":"ContainerStarted","Data":"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366"} Oct 30 07:45:21.238753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.238628 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" event={"ID":"542c41c5-8a38-41c3-824f-025144acfc4f","Type":"ContainerStarted","Data":"4a2165deeaf4fdeaa5f302cc98ce595c582830598079966ead3ce5ba071455e7"} Oct 30 07:45:21.238753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.238714 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:21.240851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.240226 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" Oct 30 07:45:21.240851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.240253 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq" event={"ID":"cad6535c-446f-4f3c-b5b3-99262e02d40b","Type":"ContainerDied","Data":"433b61d96cf4a980a4cd0ed9ef2fe5dccf0166d9bb299acb42afd4ba533bdeee"} Oct 30 07:45:21.245079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.245055 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerStarted","Data":"f61b533c153b859b420ba5e2fb2075d5fc634df566eda6da5eec39be0b3d010a"} Oct 30 07:45:21.250324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.250300 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" Oct 30 07:45:21.250453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.250295 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq" event={"ID":"d4667d03-9bf5-48ab-b817-23f8c241c7bf","Type":"ContainerDied","Data":"64fb286251fe2e0276baf3e05ff20e67819b52138edc98d0713b16fa8faa2b7f"} Oct 30 07:45:21.252747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.252718 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" event={"ID":"a9d5f6ca-4068-4c97-bbef-18b53675ff62","Type":"ContainerStarted","Data":"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed"} Oct 30 07:45:21.252879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.252770 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" event={"ID":"a9d5f6ca-4068-4c97-bbef-18b53675ff62","Type":"ContainerStarted","Data":"d1556cd895c579e50222962e4f53d2cd3c9d713cce559950c0bcfe0d8087e2d3"} Oct 30 07:45:21.252984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.252964 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:21.256189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.256162 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerStarted","Data":"cf74fdede1727503b2ffa7f7ddb76dc5c2c10de844e66514e3695c8f86306c2e"} Oct 30 07:45:21.424529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.424495 5894 scope.go:122] "RemoveContainer" containerID="c7f7aaa70bd784695940fef485d22cc2e3054fe2377ca8a7b7056f99765c9610" Oct 30 07:45:21.461603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.461548 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" podStartSLOduration=2.461533632 podStartE2EDuration="2.461533632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:21.457070249 +0000 UTC m=+2885.147987687" watchObservedRunningTime="2026-10-30 07:45:21.461533632 +0000 UTC m=+2885.152451066" Oct 30 07:45:21.549233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.549173 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" podStartSLOduration=2.5491564650000003 podStartE2EDuration="2.549156465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:45:21.544017655 +0000 UTC m=+2885.234935091" watchObservedRunningTime="2026-10-30 07:45:21.549156465 +0000 UTC m=+2885.240073902" Oct 30 07:45:21.625017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.624995 5894 scope.go:122] "RemoveContainer" containerID="3e31f3faa8e8ceddbf0c69bf73dd6c10ae0143227fd7fac8af1a6aefc4a0d06d" Oct 30 07:45:21.633588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.633561 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:21.647535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.647506 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:21.683656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.680016 5894 scope.go:122] "RemoveContainer" containerID="afb473c64f4e1632aa3edcbee304bb17758f0756cd3deb22e4d85be852275406" Oct 30 07:45:21.707124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.695476 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\" (UniqueName: \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\") pod \"fbf4fb4b-e362-4cae-8609-5557251e494d\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " Oct 30 07:45:21.707124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.705148 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\") pod \"fbf4fb4b-e362-4cae-8609-5557251e494d\" (UID: \"fbf4fb4b-e362-4cae-8609-5557251e494d\") " Oct 30 07:45:21.707124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.707063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca" pod "fbf4fb4b-e362-4cae-8609-5557251e494d" (UID: "fbf4fb4b-e362-4cae-8609-5557251e494d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:21.709164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.709141 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:21.732859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.732818 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9" pod "fbf4fb4b-e362-4cae-8609-5557251e494d" (UID: "fbf4fb4b-e362-4cae-8609-5557251e494d"). InnerVolumeSpecName "kube-api-access-g6tj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:21.781170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.781148 5894 scope.go:122] "RemoveContainer" containerID="4dc3e40c7f3075c14e4f023bec1653a51537a48ba462165e9e5db5f74dace4f9" Oct 30 07:45:21.807981 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.807948 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\") pod \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " Oct 30 07:45:21.808182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.808032 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\" (UniqueName: \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\") pod \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\" (UID: \"38827e8c-bde0-4815-a8a0-a5b60c93a64f\") " Oct 30 07:45:21.808318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.808294 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca" pod "38827e8c-bde0-4815-a8a0-a5b60c93a64f" (UID: "38827e8c-bde0-4815-a8a0-a5b60c93a64f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:21.808604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.808588 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38827e8c-bde0-4815-a8a0-a5b60c93a64f-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.808682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.808608 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4fb4b-e362-4cae-8609-5557251e494d-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.808682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.808618 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6tj9\" (UniqueName: \"kubernetes.io/projected/fbf4fb4b-e362-4cae-8609-5557251e494d-kube-api-access-g6tj9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.810544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.810515 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj" pod "38827e8c-bde0-4815-a8a0-a5b60c93a64f" (UID: "38827e8c-bde0-4815-a8a0-a5b60c93a64f"). InnerVolumeSpecName "kube-api-access-bcqxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:21.909862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.909830 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqxj\" (UniqueName: \"kubernetes.io/projected/38827e8c-bde0-4815-a8a0-a5b60c93a64f-kube-api-access-bcqxj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:21.975849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.975804 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:21.994479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:21.994447 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:22.045827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.045751 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889105-krmlv"] Oct 30 07:45:22.058516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.058469 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:22.065884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.065854 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889105-lcmhr"] Oct 30 07:45:22.089742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.089619 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:22.094210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.094186 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-bf563a-29889105-5sjbq"] Oct 30 07:45:22.126486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.126452 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:22.131099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.131030 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889105-r58rq"] Oct 30 07:45:22.269386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.269350 5894 generic.go:356] "Generic (PLEG): container finished" podID="5125cf24-506e-47c7-85f9-ff51fa746129" containerID="db2c0a3ea7c6e8a1143921664af86cd88edee2cb55b7a83c7dda4f2b02abc768" exitCode=60 Oct 30 07:45:22.269870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.269421 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerDied","Data":"db2c0a3ea7c6e8a1143921664af86cd88edee2cb55b7a83c7dda4f2b02abc768"} Oct 30 07:45:22.269870 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.269504 5894 scope.go:122] "RemoveContainer" containerID="fa5dc1e651bdc675f4c9a367d0d19fbd31734e5ea043f0c18600f3602e82e8ed" Oct 30 07:45:22.270029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.270006 5894 scope.go:122] "RemoveContainer" containerID="db2c0a3ea7c6e8a1143921664af86cd88edee2cb55b7a83c7dda4f2b02abc768" Oct 30 07:45:22.270460 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.270434 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4_odh-ai-gateway-infra(5125cf24-506e-47c7-85f9-ff51fa746129)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" Oct 30 07:45:22.271679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.271622 5894 generic.go:356] "Generic (PLEG): container finished" podID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerID="a16ba1d037d6477fd6cf1849fb8663e51ca3925cacc70fe28bda3c70d6ca700f" exitCode=60 Oct 30 07:45:22.271776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.271688 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" event={"ID":"024f2407-22e9-4298-ae1c-5c604dc4dd35","Type":"ContainerDied","Data":"a16ba1d037d6477fd6cf1849fb8663e51ca3925cacc70fe28bda3c70d6ca700f"} Oct 30 07:45:22.272121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.272102 5894 scope.go:122] "RemoveContainer" containerID="a16ba1d037d6477fd6cf1849fb8663e51ca3925cacc70fe28bda3c70d6ca700f" Oct 30 07:45:22.272481 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.272458 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg_odh-ai-gateway-infra(024f2407-22e9-4298-ae1c-5c604dc4dd35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" Oct 30 07:45:22.273211 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.273189 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" event={"ID":"38827e8c-bde0-4815-a8a0-a5b60c93a64f","Type":"ContainerDied","Data":"e486cd0acdf6dedda519afa9693347a02e931dac880613c779593a167a408e25"} Oct 30 07:45:22.273327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.273192 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d" Oct 30 07:45:22.275654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.275613 5894 generic.go:356] "Generic (PLEG): container finished" podID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerID="f61b533c153b859b420ba5e2fb2075d5fc634df566eda6da5eec39be0b3d010a" exitCode=60 Oct 30 07:45:22.275756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.275735 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerDied","Data":"f61b533c153b859b420ba5e2fb2075d5fc634df566eda6da5eec39be0b3d010a"} Oct 30 07:45:22.276125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.276107 5894 scope.go:122] "RemoveContainer" containerID="f61b533c153b859b420ba5e2fb2075d5fc634df566eda6da5eec39be0b3d010a" Oct 30 07:45:22.276456 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.276430 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5_odh-ai-gateway-infra(bdd566e3-dde0-400a-a58a-5c6e37ad55ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" Oct 30 07:45:22.277308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.277291 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" Oct 30 07:45:22.277416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.277310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc" event={"ID":"fbf4fb4b-e362-4cae-8609-5557251e494d","Type":"ContainerDied","Data":"46f1bb996e5500e245e2a47196a7945342667e64510d0238830d6d787167bc2a"} Oct 30 07:45:22.280856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.280831 5894 generic.go:356] "Generic (PLEG): container finished" podID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerID="cf74fdede1727503b2ffa7f7ddb76dc5c2c10de844e66514e3695c8f86306c2e" exitCode=60 Oct 30 07:45:22.280979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.280899 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerDied","Data":"cf74fdede1727503b2ffa7f7ddb76dc5c2c10de844e66514e3695c8f86306c2e"} Oct 30 07:45:22.281292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.281272 5894 scope.go:122] "RemoveContainer" containerID="cf74fdede1727503b2ffa7f7ddb76dc5c2c10de844e66514e3695c8f86306c2e" Oct 30 07:45:22.281608 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.281584 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h_odh-ai-gateway-infra(4980eabc-1453-40d4-92b3-b3e8f489e1a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" Oct 30 07:45:22.284736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.284716 5894 generic.go:356] "Generic (PLEG): container finished" podID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerID="7801e1db0ced34265e2defd6255a51356eeaf4dcf014217c41a63de783c751e5" exitCode=60 Oct 30 07:45:22.284853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.284798 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerDied","Data":"7801e1db0ced34265e2defd6255a51356eeaf4dcf014217c41a63de783c751e5"} Oct 30 07:45:22.285331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.285305 5894 scope.go:122] "RemoveContainer" containerID="7801e1db0ced34265e2defd6255a51356eeaf4dcf014217c41a63de783c751e5" Oct 30 07:45:22.285619 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.285594 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj_odh-ai-gateway-infra(8f6d77cd-1723-4b98-a138-d3cfc8de9b62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" Oct 30 07:45:22.287290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.287198 5894 scope.go:122] "RemoveContainer" containerID="fe079f57046aa777b9d34882479117a7024b18d250c2250ce9a694640d955e7e" Oct 30 07:45:22.287290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.287264 5894 generic.go:356] "Generic (PLEG): container finished" podID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerID="a73731e1bae4c40b0e44e70fd6b9515f2837c47f2cec3cd9f74179541aedf116" exitCode=60 Oct 30 07:45:22.287426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.287347 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerDied","Data":"a73731e1bae4c40b0e44e70fd6b9515f2837c47f2cec3cd9f74179541aedf116"} Oct 30 07:45:22.291121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.291097 5894 generic.go:356] "Generic (PLEG): container finished" podID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerID="bbb8ba66d704564112062db53f1a111162b043f7d1fe001c26bb3577b0784cca" exitCode=60 Oct 30 07:45:22.291223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.291160 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerDied","Data":"bbb8ba66d704564112062db53f1a111162b043f7d1fe001c26bb3577b0784cca"} Oct 30 07:45:22.291562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.291544 5894 scope.go:122] "RemoveContainer" containerID="bbb8ba66d704564112062db53f1a111162b043f7d1fe001c26bb3577b0784cca" Oct 30 07:45:22.291913 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.291893 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd_odh-ai-gateway-infra(89601e6b-88f3-4a7a-9d88-cc02c7c9b67c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" Oct 30 07:45:22.293224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.293206 5894 generic.go:356] "Generic (PLEG): container finished" podID="22679da9-0561-4760-a07b-6adff3f83ef0" containerID="ceb95d73786675c0772836c164bd44470a4c12fa9175b741dd2c20c927d53e4a" exitCode=60 Oct 30 07:45:22.293327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.293273 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerDied","Data":"ceb95d73786675c0772836c164bd44470a4c12fa9175b741dd2c20c927d53e4a"} Oct 30 07:45:22.293599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.293583 5894 scope.go:122] "RemoveContainer" containerID="ceb95d73786675c0772836c164bd44470a4c12fa9175b741dd2c20c927d53e4a" Oct 30 07:45:22.293933 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.293912 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d_odh-ai-gateway-infra(22679da9-0561-4760-a07b-6adff3f83ef0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" Oct 30 07:45:22.295319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.295294 5894 generic.go:356] "Generic (PLEG): container finished" podID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerID="3a0b3b63d4b780959e6214042ecf82bf6ee32aa99675547cd5860e6c68bdc7fd" exitCode=60 Oct 30 07:45:22.295425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.295359 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerDied","Data":"3a0b3b63d4b780959e6214042ecf82bf6ee32aa99675547cd5860e6c68bdc7fd"} Oct 30 07:45:22.295710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.295693 5894 scope.go:122] "RemoveContainer" containerID="3a0b3b63d4b780959e6214042ecf82bf6ee32aa99675547cd5860e6c68bdc7fd" Oct 30 07:45:22.295979 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.295959 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd_odh-ai-gateway-infra(c895d4bf-e5f2-44cb-959f-d36c5b14e623)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" Oct 30 07:45:22.297926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.297901 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerStarted","Data":"42b2529a0139242f79233a2c47520ba8e9e066bf93deb31c3d455dbdd880a288"} Oct 30 07:45:22.300994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.300967 5894 generic.go:356] "Generic (PLEG): container finished" podID="6e4cceaa-bc57-4862-8212-f6920f930785" containerID="5082e81ad789fbfb9eba6896380658e7ee20b3c2f4ba2576bfc68958bc671ba9" exitCode=60 Oct 30 07:45:22.301102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.301075 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" event={"ID":"6e4cceaa-bc57-4862-8212-f6920f930785","Type":"ContainerDied","Data":"5082e81ad789fbfb9eba6896380658e7ee20b3c2f4ba2576bfc68958bc671ba9"} Oct 30 07:45:22.301435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.301417 5894 scope.go:122] "RemoveContainer" containerID="5082e81ad789fbfb9eba6896380658e7ee20b3c2f4ba2576bfc68958bc671ba9" Oct 30 07:45:22.301742 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:22.301715 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9_odh-ai-gateway-infra(6e4cceaa-bc57-4862-8212-f6920f930785)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" Oct 30 07:45:22.305296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.305280 5894 scope.go:122] "RemoveContainer" containerID="525b54904e34912d8f6524da9e5c428a517a09d2f1456a75fcd65c995a1c071f" Oct 30 07:45:22.308549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.308524 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:22.317284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.317260 5894 scope.go:122] "RemoveContainer" containerID="ad603db6c456ccf1afba286f9121ebc90e015cea04d51aeb38ed89bdba6e4e98" Oct 30 07:45:22.335949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.335918 5894 scope.go:122] "RemoveContainer" containerID="e0047bbac2ce1347c83d6661cdda475b4be8ab77b3dc9fa0af3ad94955187520" Oct 30 07:45:22.350177 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.350153 5894 scope.go:122] "RemoveContainer" containerID="2e564d658a6dfa8c002f197debb9b8708a5a047c339e752c634e684f8e284f50" Oct 30 07:45:22.350521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.350494 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:22.380824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.380777 5894 scope.go:122] "RemoveContainer" containerID="2d140f239d237499fa740ed993019af94c42dd57d28bfa1bc450bd24f777ee6d" Oct 30 07:45:22.411007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.410975 5894 scope.go:122] "RemoveContainer" containerID="15f159d6caa4ee65f754f21a36019348285f9897d3e5b8238a4aeddf39b5eadf" Oct 30 07:45:22.429716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.429601 5894 scope.go:122] "RemoveContainer" containerID="81a906284fa8cddb05e73bc98d1043f0389a2ef990d64ec58059ee69184618fa" Oct 30 07:45:22.439410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.439377 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de640e8-db88-438e-8279-e7efd370deb6" path="/var/lib/kubelet/pods/1de640e8-db88-438e-8279-e7efd370deb6/volumes" Oct 30 07:45:22.440089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.440063 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1666912-5749-4286-bb70-75f9968f3cca" path="/var/lib/kubelet/pods/a1666912-5749-4286-bb70-75f9968f3cca/volumes" Oct 30 07:45:22.440727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.440704 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" path="/var/lib/kubelet/pods/cad6535c-446f-4f3c-b5b3-99262e02d40b/volumes" Oct 30 07:45:22.441751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.441727 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" path="/var/lib/kubelet/pods/d4667d03-9bf5-48ab-b817-23f8c241c7bf/volumes" Oct 30 07:45:22.442345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.442325 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" path="/var/lib/kubelet/pods/fde8b11e-629f-488c-82a3-90ab92d49c4f/volumes" Oct 30 07:45:22.449507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.449484 5894 scope.go:122] "RemoveContainer" containerID="0a862868ad688f07999a3fb110825da9498db3d6f21fc52b2f2fa422ef597e34" Oct 30 07:45:22.460110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.460082 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:22.466094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.466076 5894 scope.go:122] "RemoveContainer" containerID="35fec1e10fb778fd77c397b067d0364872f13b28055429ab00d26f50b614a775" Oct 30 07:45:22.480899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.480871 5894 scope.go:122] "RemoveContainer" containerID="2282bc0a53030853bbcb20cc25ad0819fa251a29ecce1bd8b71aba9c5a73760d" Oct 30 07:45:22.532127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.532097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\" (UniqueName: \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\") pod \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " Oct 30 07:45:22.532292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.532241 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\") pod \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\" (UID: \"c4614d8a-8a8d-49d7-a91b-2c986228b6a6\") " Oct 30 07:45:22.532603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.532572 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca" pod "c4614d8a-8a8d-49d7-a91b-2c986228b6a6" (UID: "c4614d8a-8a8d-49d7-a91b-2c986228b6a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:22.533028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.532958 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:22.534442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.534420 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t" pod "c4614d8a-8a8d-49d7-a91b-2c986228b6a6" (UID: "c4614d8a-8a8d-49d7-a91b-2c986228b6a6"). InnerVolumeSpecName "kube-api-access-qtn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:22.580310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.580264 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:22.585381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.585351 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889105-7jgvc"] Oct 30 07:45:22.601349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.601315 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:22.634863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.634813 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtn7t\" (UniqueName: \"kubernetes.io/projected/c4614d8a-8a8d-49d7-a91b-2c986228b6a6-kube-api-access-qtn7t\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:22.703598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.703567 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:22.708148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.708118 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889105-w8s6d"] Oct 30 07:45:22.731304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.731276 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:22.783714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.783677 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:22.948054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:22.947966 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:23.325278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.325252 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" Oct 30 07:45:23.325954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.325919 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv" event={"ID":"c4614d8a-8a8d-49d7-a91b-2c986228b6a6","Type":"ContainerDied","Data":"ebb30a807328ddd4a39e450a64755b23e80fe80dcbadae528696ea70a11b597d"} Oct 30 07:45:23.326085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.326021 5894 scope.go:122] "RemoveContainer" containerID="a73731e1bae4c40b0e44e70fd6b9515f2837c47f2cec3cd9f74179541aedf116" Oct 30 07:45:23.336713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.335130 5894 generic.go:356] "Generic (PLEG): container finished" podID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerID="42b2529a0139242f79233a2c47520ba8e9e066bf93deb31c3d455dbdd880a288" exitCode=60 Oct 30 07:45:23.336713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.335233 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerDied","Data":"42b2529a0139242f79233a2c47520ba8e9e066bf93deb31c3d455dbdd880a288"} Oct 30 07:45:23.347048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.347015 5894 scope.go:122] "RemoveContainer" containerID="c7f7aaa70bd784695940fef485d22cc2e3054fe2377ca8a7b7056f99765c9610" Oct 30 07:45:23.388344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.376530 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:23.620768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.620730 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:23.626218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.626186 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2e8b2a-29889105-qgcmv"] Oct 30 07:45:23.646725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.646680 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:23.651957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.651925 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:23.653296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.653268 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:23.703407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.703074 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:23.784771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.784735 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\" (UniqueName: \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\") pod \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " Oct 30 07:45:23.784959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.784902 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\") pod \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " Oct 30 07:45:23.785051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.785030 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\") pod \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\" (UID: \"4980eabc-1453-40d4-92b3-b3e8f489e1a2\") " Oct 30 07:45:23.785227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.785205 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\" (UniqueName: \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\") pod \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\" (UID: \"bdd566e3-dde0-400a-a58a-5c6e37ad55ad\") " Oct 30 07:45:23.786573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.786542 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca" pod "bdd566e3-dde0-400a-a58a-5c6e37ad55ad" (UID: "bdd566e3-dde0-400a-a58a-5c6e37ad55ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:23.788182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.788134 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca" pod "4980eabc-1453-40d4-92b3-b3e8f489e1a2" (UID: "4980eabc-1453-40d4-92b3-b3e8f489e1a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:23.789973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.789947 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8" pod "4980eabc-1453-40d4-92b3-b3e8f489e1a2" (UID: "4980eabc-1453-40d4-92b3-b3e8f489e1a2"). InnerVolumeSpecName "kube-api-access-rbmn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:23.790372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.790347 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p" pod "bdd566e3-dde0-400a-a58a-5c6e37ad55ad" (UID: "bdd566e3-dde0-400a-a58a-5c6e37ad55ad"). InnerVolumeSpecName "kube-api-access-s9p4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:23.828994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.828366 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:23.831118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.831087 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:23.834952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.834931 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:23.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.838553 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:23.841483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.841456 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:23.843435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.843408 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:23.843435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.843424 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:23.889949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.889815 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9p4p\" (UniqueName: \"kubernetes.io/projected/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-kube-api-access-s9p4p\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.889949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.889857 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmn8\" (UniqueName: \"kubernetes.io/projected/4980eabc-1453-40d4-92b3-b3e8f489e1a2-kube-api-access-rbmn8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.889949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.889875 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd566e3-dde0-400a-a58a-5c6e37ad55ad-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.889949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.889890 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4980eabc-1453-40d4-92b3-b3e8f489e1a2-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991087 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\") pod \"a4eced12-8c18-4a34-9c19-ea57979730cb\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991174 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\" (UniqueName: \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\") pod \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991206 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\") pod \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991281 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\") pod \"22679da9-0561-4760-a07b-6adff3f83ef0\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991361 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\" (UniqueName: \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\") pod \"5125cf24-506e-47c7-85f9-ff51fa746129\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991415 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\" (UniqueName: \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\") pod \"a4eced12-8c18-4a34-9c19-ea57979730cb\" (UID: \"a4eced12-8c18-4a34-9c19-ea57979730cb\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991455 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\") pod \"5125cf24-506e-47c7-85f9-ff51fa746129\" (UID: \"5125cf24-506e-47c7-85f9-ff51fa746129\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991515 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\") pod \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\" (UID: \"c895d4bf-e5f2-44cb-959f-d36c5b14e623\") " Oct 30 07:45:23.993338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.991552 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\" (UniqueName: \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\") pod \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\" (UID: \"8f6d77cd-1723-4b98-a138-d3cfc8de9b62\") " Oct 30 07:45:23.995414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.994183 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\" (UniqueName: \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\") pod \"22679da9-0561-4760-a07b-6adff3f83ef0\" (UID: \"22679da9-0561-4760-a07b-6adff3f83ef0\") " Oct 30 07:45:23.998906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:23.998598 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca" pod "5125cf24-506e-47c7-85f9-ff51fa746129" (UID: "5125cf24-506e-47c7-85f9-ff51fa746129"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.004359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.004051 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca" pod "c895d4bf-e5f2-44cb-959f-d36c5b14e623" (UID: "c895d4bf-e5f2-44cb-959f-d36c5b14e623"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.006500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.006176 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca" pod "a4eced12-8c18-4a34-9c19-ea57979730cb" (UID: "a4eced12-8c18-4a34-9c19-ea57979730cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.007132 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.006889 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca" pod "8f6d77cd-1723-4b98-a138-d3cfc8de9b62" (UID: "8f6d77cd-1723-4b98-a138-d3cfc8de9b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.007893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.007468 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca" pod "22679da9-0561-4760-a07b-6adff3f83ef0" (UID: "22679da9-0561-4760-a07b-6adff3f83ef0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.007893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.007814 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8" pod "c895d4bf-e5f2-44cb-959f-d36c5b14e623" (UID: "c895d4bf-e5f2-44cb-959f-d36c5b14e623"). InnerVolumeSpecName "kube-api-access-4hcm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.010984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.010945 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks" pod "8f6d77cd-1723-4b98-a138-d3cfc8de9b62" (UID: "8f6d77cd-1723-4b98-a138-d3cfc8de9b62"). InnerVolumeSpecName "kube-api-access-jlvks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.022895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.021883 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l" pod "22679da9-0561-4760-a07b-6adff3f83ef0" (UID: "22679da9-0561-4760-a07b-6adff3f83ef0"). InnerVolumeSpecName "kube-api-access-fp45l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.026142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.026099 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt" pod "a4eced12-8c18-4a34-9c19-ea57979730cb" (UID: "a4eced12-8c18-4a34-9c19-ea57979730cb"). InnerVolumeSpecName "kube-api-access-279xt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.037918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.037865 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n" pod "5125cf24-506e-47c7-85f9-ff51fa746129" (UID: "5125cf24-506e-47c7-85f9-ff51fa746129"). InnerVolumeSpecName "kube-api-access-zt48n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.061097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.061061 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:24.075114 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.075029 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:24.097227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097130 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-279xt\" (UniqueName: \"kubernetes.io/projected/a4eced12-8c18-4a34-9c19-ea57979730cb-kube-api-access-279xt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097421 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5125cf24-506e-47c7-85f9-ff51fa746129-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097453 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097454 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c895d4bf-e5f2-44cb-959f-d36c5b14e623-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097471 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlvks\" (UniqueName: \"kubernetes.io/projected/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-kube-api-access-jlvks\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097486 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp45l\" (UniqueName: \"kubernetes.io/projected/22679da9-0561-4760-a07b-6adff3f83ef0-kube-api-access-fp45l\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097501 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4eced12-8c18-4a34-9c19-ea57979730cb-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097516 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hcm8\" (UniqueName: \"kubernetes.io/projected/c895d4bf-e5f2-44cb-959f-d36c5b14e623-kube-api-access-4hcm8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097531 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6d77cd-1723-4b98-a138-d3cfc8de9b62-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097546 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22679da9-0561-4760-a07b-6adff3f83ef0-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.097733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.097559 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt48n\" (UniqueName: \"kubernetes.io/projected/5125cf24-506e-47c7-85f9-ff51fa746129-kube-api-access-zt48n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.153113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.153083 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:24.198692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.198650 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\") pod \"024f2407-22e9-4298-ae1c-5c604dc4dd35\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " Oct 30 07:45:24.198901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.198756 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\") pod \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " Oct 30 07:45:24.198901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.198877 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\" (UniqueName: \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\") pod \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\" (UID: \"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c\") " Oct 30 07:45:24.199043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.198908 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\" (UniqueName: \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\") pod \"024f2407-22e9-4298-ae1c-5c604dc4dd35\" (UID: \"024f2407-22e9-4298-ae1c-5c604dc4dd35\") " Oct 30 07:45:24.199043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.199031 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca" pod "024f2407-22e9-4298-ae1c-5c604dc4dd35" (UID: "024f2407-22e9-4298-ae1c-5c604dc4dd35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.199193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.199165 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca" pod "89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" (UID: "89601e6b-88f3-4a7a-9d88-cc02c7c9b67c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.199647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.199606 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/024f2407-22e9-4298-ae1c-5c604dc4dd35-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.199751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.199650 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.201296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.201261 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm" pod "024f2407-22e9-4298-ae1c-5c604dc4dd35" (UID: "024f2407-22e9-4298-ae1c-5c604dc4dd35"). InnerVolumeSpecName "kube-api-access-9l2zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.201893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.201871 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr" pod "89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" (UID: "89601e6b-88f3-4a7a-9d88-cc02c7c9b67c"). InnerVolumeSpecName "kube-api-access-4fzdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.300715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.300664 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\") pod \"6e4cceaa-bc57-4862-8212-f6920f930785\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " Oct 30 07:45:24.300918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.300869 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\" (UniqueName: \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\") pod \"6e4cceaa-bc57-4862-8212-f6920f930785\" (UID: \"6e4cceaa-bc57-4862-8212-f6920f930785\") " Oct 30 07:45:24.301549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.301132 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca" pod "6e4cceaa-bc57-4862-8212-f6920f930785" (UID: "6e4cceaa-bc57-4862-8212-f6920f930785"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:45:24.301751 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.301731 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l2zm\" (UniqueName: \"kubernetes.io/projected/024f2407-22e9-4298-ae1c-5c604dc4dd35-kube-api-access-9l2zm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.301825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.301756 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e4cceaa-bc57-4862-8212-f6920f930785-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.301825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.301773 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzdr\" (UniqueName: \"kubernetes.io/projected/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c-kube-api-access-4fzdr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.304195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.304145 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw" pod "6e4cceaa-bc57-4862-8212-f6920f930785" (UID: "6e4cceaa-bc57-4862-8212-f6920f930785"). InnerVolumeSpecName "kube-api-access-fw9lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:45:24.356135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.356056 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" Oct 30 07:45:24.356805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.356069 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg" event={"ID":"024f2407-22e9-4298-ae1c-5c604dc4dd35","Type":"ContainerDied","Data":"5cb185bb0467be1f15b6c8fd2dc9966aa8cd45ca169d65982be7968396bb725e"} Oct 30 07:45:24.356905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.356857 5894 scope.go:122] "RemoveContainer" containerID="a16ba1d037d6477fd6cf1849fb8663e51ca3925cacc70fe28bda3c70d6ca700f" Oct 30 07:45:24.358289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.358259 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" event={"ID":"bdd566e3-dde0-400a-a58a-5c6e37ad55ad","Type":"ContainerDied","Data":"2e63c8eb3be48801fc38781e9b8aec373a9d0b959d91166f33ac253a62911ef4"} Oct 30 07:45:24.358421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.358348 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5" Oct 30 07:45:24.396320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.396277 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" Oct 30 07:45:24.396792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.396285 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h" event={"ID":"4980eabc-1453-40d4-92b3-b3e8f489e1a2","Type":"ContainerDied","Data":"cb57790680d2b751d47de927a14d0196b5212dda4fbb733364c3a6af52c1926f"} Oct 30 07:45:24.399463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.399393 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" event={"ID":"8f6d77cd-1723-4b98-a138-d3cfc8de9b62","Type":"ContainerDied","Data":"e0827612ba5be56b1e990f228c75db6276ad3aa0374aa19b2dba7f544ad45277"} Oct 30 07:45:24.399603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.399585 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj" Oct 30 07:45:24.404646 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.404600 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9lw\" (UniqueName: \"kubernetes.io/projected/6e4cceaa-bc57-4862-8212-f6920f930785-kube-api-access-fw9lw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:45:24.423672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.423400 5894 scope.go:122] "RemoveContainer" containerID="f61b533c153b859b420ba5e2fb2075d5fc634df566eda6da5eec39be0b3d010a" Oct 30 07:45:24.425150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.425117 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" event={"ID":"89601e6b-88f3-4a7a-9d88-cc02c7c9b67c","Type":"ContainerDied","Data":"cb1fd08098efd1d314c7bf2a887c604eca0335b4f6cae8c77b2b898a0bd1d55b"} Oct 30 07:45:24.425269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.425207 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd" Oct 30 07:45:24.443088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.443053 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" Oct 30 07:45:24.451747 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.451720 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" Oct 30 07:45:24.453855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.453828 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" Oct 30 07:45:24.454247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.454200 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" path="/var/lib/kubelet/pods/38827e8c-bde0-4815-a8a0-a5b60c93a64f/volumes" Oct 30 07:45:24.455187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.455159 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" path="/var/lib/kubelet/pods/c4614d8a-8a8d-49d7-a91b-2c986228b6a6/volumes" Oct 30 07:45:24.455919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.455897 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" path="/var/lib/kubelet/pods/fbf4fb4b-e362-4cae-8609-5557251e494d/volumes" Oct 30 07:45:24.456721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.456699 5894 scope.go:122] "RemoveContainer" containerID="cf74fdede1727503b2ffa7f7ddb76dc5c2c10de844e66514e3695c8f86306c2e" Oct 30 07:45:24.457415 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.457395 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" Oct 30 07:45:24.463974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.458023 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d" event={"ID":"22679da9-0561-4760-a07b-6adff3f83ef0","Type":"ContainerDied","Data":"9811c3696453d3137b3adf8ac1e516585256e80baadb02b2d879994aafcf6b3d"} Oct 30 07:45:24.464126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464111 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:24.464247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464230 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-bb2e8f-29889105-j24vg"] Oct 30 07:45:24.464324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464263 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd" event={"ID":"c895d4bf-e5f2-44cb-959f-d36c5b14e623","Type":"ContainerDied","Data":"2b069415c057d97a82c65d776fd0fdf047b076f53e9e1f2f6c0f805492d86e04"} Oct 30 07:45:24.464324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4" event={"ID":"a4eced12-8c18-4a34-9c19-ea57979730cb","Type":"ContainerDied","Data":"3d52c76763951dbe16a9d2c508e83dde5961a1bdf879fccf7afd1bf100f21f34"} Oct 30 07:45:24.464446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464331 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9" event={"ID":"6e4cceaa-bc57-4862-8212-f6920f930785","Type":"ContainerDied","Data":"3cadeb8db3868ed1ffa03dfb574439e87ca1a2b1ac1514da263e576fce5a3261"} Oct 30 07:45:24.464854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464833 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" event={"ID":"5125cf24-506e-47c7-85f9-ff51fa746129","Type":"ContainerDied","Data":"da463eaedf585a23af11656af0616ebedf90541103df67714fb372e8c358ffc1"} Oct 30 07:45:24.464943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.464932 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4" Oct 30 07:45:24.481904 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.481879 5894 scope.go:122] "RemoveContainer" containerID="7801e1db0ced34265e2defd6255a51356eeaf4dcf014217c41a63de783c751e5" Oct 30 07:45:24.492849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.492830 5894 scope.go:122] "RemoveContainer" containerID="bbb8ba66d704564112062db53f1a111162b043f7d1fe001c26bb3577b0784cca" Oct 30 07:45:24.504093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.504071 5894 scope.go:122] "RemoveContainer" containerID="ceb95d73786675c0772836c164bd44470a4c12fa9175b741dd2c20c927d53e4a" Oct 30 07:45:24.504940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.504920 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:24.509590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.509559 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889105-lp4lj"] Oct 30 07:45:24.515036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.515017 5894 scope.go:122] "RemoveContainer" containerID="3a0b3b63d4b780959e6214042ecf82bf6ee32aa99675547cd5860e6c68bdc7fd" Oct 30 07:45:24.525443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.525416 5894 scope.go:122] "RemoveContainer" containerID="42b2529a0139242f79233a2c47520ba8e9e066bf93deb31c3d455dbdd880a288" Oct 30 07:45:24.536457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.536433 5894 scope.go:122] "RemoveContainer" containerID="5082e81ad789fbfb9eba6896380658e7ee20b3c2f4ba2576bfc68958bc671ba9" Oct 30 07:45:24.542654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.542607 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:24.546830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.546802 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889105-7zb4h"] Oct 30 07:45:24.547917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.547900 5894 scope.go:122] "RemoveContainer" containerID="db2c0a3ea7c6e8a1143921664af86cd88edee2cb55b7a83c7dda4f2b02abc768" Oct 30 07:45:24.582697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.582660 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:24.587524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.587501 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889105-crjjd"] Oct 30 07:45:24.622412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.622326 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:24.627849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.627819 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-d89593-29889105-ms7r5"] Oct 30 07:45:24.665608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.665571 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:24.669607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.669583 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889105-jrz5d"] Oct 30 07:45:24.703539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.703505 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:24.709323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.709293 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889105-56sf4"] Oct 30 07:45:24.747203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.747165 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:24.754190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.754158 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-2761aa-29889105-dv9kd"] Oct 30 07:45:24.788307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.788269 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:24.794717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.794684 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889105-spgf4"] Oct 30 07:45:24.830050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.830007 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:24.836356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:24.836319 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a0b3f5-29889105-hbks9"] Oct 30 07:45:25.627146 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:25.627115 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:26.434092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.434060 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" path="/var/lib/kubelet/pods/024f2407-22e9-4298-ae1c-5c604dc4dd35/volumes" Oct 30 07:45:26.434624 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.434599 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" path="/var/lib/kubelet/pods/22679da9-0561-4760-a07b-6adff3f83ef0/volumes" Oct 30 07:45:26.435214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.435196 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" path="/var/lib/kubelet/pods/4980eabc-1453-40d4-92b3-b3e8f489e1a2/volumes" Oct 30 07:45:26.436027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.436008 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" path="/var/lib/kubelet/pods/5125cf24-506e-47c7-85f9-ff51fa746129/volumes" Oct 30 07:45:26.436409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.436397 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" path="/var/lib/kubelet/pods/6e4cceaa-bc57-4862-8212-f6920f930785/volumes" Oct 30 07:45:26.436811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.436798 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" path="/var/lib/kubelet/pods/89601e6b-88f3-4a7a-9d88-cc02c7c9b67c/volumes" Oct 30 07:45:26.437480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.437465 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" path="/var/lib/kubelet/pods/8f6d77cd-1723-4b98-a138-d3cfc8de9b62/volumes" Oct 30 07:45:26.437936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.437919 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" path="/var/lib/kubelet/pods/a4eced12-8c18-4a34-9c19-ea57979730cb/volumes" Oct 30 07:45:26.438310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.438297 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" path="/var/lib/kubelet/pods/bdd566e3-dde0-400a-a58a-5c6e37ad55ad/volumes" Oct 30 07:45:26.438988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:26.438973 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" path="/var/lib/kubelet/pods/c895d4bf-e5f2-44cb-959f-d36c5b14e623/volumes" Oct 30 07:45:27.306346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:27.306307 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:28.963766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:28.963737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:30.641505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:30.641475 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:31.051306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:31.051268 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:31.171790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:31.171751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:32.304982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:32.304949 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:45:32.305463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:32.305030 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:45:32.322890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:32.322847 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:34.038738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:34.038698 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:34.836444 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:45:34.836400 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:45:35.723502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:35.723466 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:37.284242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:37.284200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:37.440279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:37.440236 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:38.157273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:38.157175 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:39.411156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:39.411116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:41.126708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:41.126666 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:42.733864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:42.733827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:44.434404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:44.434364 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:46.018125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:46.018063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:47.690780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:47.690734 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:49.483118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:49.483079 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:51.116532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:51.116494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:52.838766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:52.838725 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:54.770322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:54.770282 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:56.419574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:56.419532 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:56.545043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:56.545007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:56.867863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:56.867811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:57.345230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:57.345193 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:57.483660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:57.481352 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:58.108207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:58.108177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:45:59.947730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:45:59.947696 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:00.912066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:00.906978 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:46:00.929657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:00.918975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:01.050770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:01.050732 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:02.126667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.123404 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:02.743800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.743761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:02.853875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.853840 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:02.928234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.928202 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:46:02.928995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.928980 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.928997 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929009 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929014 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929020 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929026 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929035 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929040 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.929046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929045 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929050 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929059 5894 cpu_manager.go:414] "containerMap: removing container" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929063 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929070 5894 cpu_manager.go:414] "containerMap: removing container" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929074 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929083 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929088 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929096 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929101 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929107 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929111 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929119 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929123 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929130 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929135 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929149 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929153 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929160 5894 cpu_manager.go:414] "containerMap: removing container" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929164 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929170 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929175 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929184 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929189 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929196 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929200 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929206 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929210 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929221 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929225 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929232 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929236 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929242 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929247 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929252 5894 cpu_manager.go:414] "containerMap: removing container" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929256 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929265 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929269 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929275 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929280 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929285 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929290 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929300 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929305 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929311 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929315 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929321 5894 cpu_manager.go:414] "containerMap: removing container" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929326 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929331 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.929313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929336 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929342 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929347 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929353 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929357 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929364 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929368 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929375 5894 cpu_manager.go:414] "containerMap: removing container" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929379 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929390 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929394 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929402 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929406 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929413 5894 cpu_manager.go:414] "containerMap: removing container" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929417 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929423 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929427 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929433 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929437 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929449 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929453 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929460 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929464 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929475 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929479 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929487 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929491 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929498 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929503 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929508 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929512 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929517 5894 cpu_manager.go:414] "containerMap: removing container" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929521 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929647 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929661 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929671 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929677 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929683 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929690 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929698 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929703 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929709 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929714 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929719 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929725 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929732 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929737 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929745 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929751 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929758 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5125cf24-506e-47c7-85f9-ff51fa746129" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929764 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929770 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929778 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929784 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929790 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929797 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929801 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929807 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4614d8a-8a8d-49d7-a91b-2c986228b6a6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929814 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929819 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929824 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929831 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929838 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929844 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929851 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929855 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929863 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929869 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929874 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929879 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929886 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929892 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="22679da9-0561-4760-a07b-6adff3f83ef0" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929898 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929904 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929909 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929914 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c895d4bf-e5f2-44cb-959f-d36c5b14e623" containerName="cleanup" Oct 30 07:46:02.931228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.929918 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:02.935454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.935436 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:02.955855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.955825 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:46:02.963009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.962971 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-63f24b-dockercfg-jtz4r\"" Oct 30 07:46:02.963179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:02.962971 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-63f24b\"" Oct 30 07:46:03.087245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.087198 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.088772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.088746 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn6wk\" (UniqueName: \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.095746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.095708 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:46:03.134658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.098075 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:03.183713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.165338 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89601e6b-88f3-4a7a-9d88-cc02c7c9b67c" containerName="cleanup" Oct 30 07:46:03.208496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.208463 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:03.208786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.208765 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:03.208990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.208977 5894 cpu_manager.go:414] "containerMap: removing container" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:03.209096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209085 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:03.209196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209187 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:03.209272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209261 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6d77cd-1723-4b98-a138-d3cfc8de9b62" containerName="cleanup" Oct 30 07:46:03.209398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209387 5894 cpu_manager.go:414] "containerMap: removing container" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:03.209473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209463 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:03.209590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209578 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:03.209695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209684 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4eced12-8c18-4a34-9c19-ea57979730cb" containerName="cleanup" Oct 30 07:46:03.209806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209797 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:03.209888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.209878 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de640e8-db88-438e-8279-e7efd370deb6" containerName="cleanup" Oct 30 07:46:03.210022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.210010 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:03.210138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.210126 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:03.210407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.210374 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:03.210566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.210532 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:03.211519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.211487 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde8b11e-629f-488c-82a3-90ab92d49c4f" containerName="cleanup" Oct 30 07:46:03.211693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.211679 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1666912-5749-4286-bb70-75f9968f3cca" containerName="cleanup" Oct 30 07:46:03.211838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.211825 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="38827e8c-bde0-4815-a8a0-a5b60c93a64f" containerName="cleanup" Oct 30 07:46:03.211927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.211917 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="024f2407-22e9-4298-ae1c-5c604dc4dd35" containerName="cleanup" Oct 30 07:46:03.212236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.212210 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4980eabc-1453-40d4-92b3-b3e8f489e1a2" containerName="cleanup" Oct 30 07:46:03.212406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.212395 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf4fb4b-e362-4cae-8609-5557251e494d" containerName="cleanup" Oct 30 07:46:03.212501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.212489 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd566e3-dde0-400a-a58a-5c6e37ad55ad" containerName="cleanup" Oct 30 07:46:03.212588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.212579 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cad6535c-446f-4f3c-b5b3-99262e02d40b" containerName="cleanup" Oct 30 07:46:03.212710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.212699 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4667d03-9bf5-48ab-b817-23f8c241c7bf" containerName="cleanup" Oct 30 07:46:03.213803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.213786 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e4cceaa-bc57-4862-8212-f6920f930785" containerName="cleanup" Oct 30 07:46:03.214122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.199303 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.215228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.198390 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.215602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.215580 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn6wk\" (UniqueName: \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.222685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.222601 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:46:03.223036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.223015 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.270190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.270155 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn6wk\" (UniqueName: \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\") pod \"payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.321468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.321428 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.321468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.321476 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwdr\" (UniqueName: \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.423831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.423726 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.423831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.423794 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwdr\" (UniqueName: \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.424658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.424614 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.447746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.447709 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwdr\" (UniqueName: \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\") pod \"payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.556268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.556231 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:03.572838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.572813 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:03.851879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.851844 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:03.871833 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:03.871797 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abc5c5a_fe25_40e2_92a9_44d8fb3bd54a.slice/crio-1eeb1c712d9154a0c5e8bdcd3effb22810ea02a59ef527b3f79c2fce3930a8d2 WatchSource:0}: Error finding container 1eeb1c712d9154a0c5e8bdcd3effb22810ea02a59ef527b3f79c2fce3930a8d2: Status 404 returned error can't find the container with id 1eeb1c712d9154a0c5e8bdcd3effb22810ea02a59ef527b3f79c2fce3930a8d2 Oct 30 07:46:03.872457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.872435 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:46:03.936611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:03.935492 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:46:03.957580 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:03.957537 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeed0dc6_9dcf_4b6f_97d2_54470dce1e39.slice/crio-282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d WatchSource:0}: Error finding container 282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d: Status 404 returned error can't find the container with id 282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d Oct 30 07:46:04.829601 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:04.829564 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:46:04.865770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.865728 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" event={"ID":"feed0dc6-9dcf-4b6f-97d2-54470dce1e39","Type":"ContainerStarted","Data":"4165841bfc3fc359b074cd7c32fd8686aa74c1ca5c7cdff249b0afe285caa958"} Oct 30 07:46:04.865988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.865789 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" event={"ID":"feed0dc6-9dcf-4b6f-97d2-54470dce1e39","Type":"ContainerStarted","Data":"282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d"} Oct 30 07:46:04.865988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.865818 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:04.867480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.867456 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" event={"ID":"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a","Type":"ContainerStarted","Data":"c8379124a295e72b929eb8ae0738d84308282d7ba6f87d6c655b3c50f8c66b5a"} Oct 30 07:46:04.867608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.867502 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" event={"ID":"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a","Type":"ContainerStarted","Data":"1eeb1c712d9154a0c5e8bdcd3effb22810ea02a59ef527b3f79c2fce3930a8d2"} Oct 30 07:46:04.867608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.867549 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:04.932658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.916396 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" podStartSLOduration=2.9163747989999997 podStartE2EDuration="2.916374799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:04.910926895 +0000 UTC m=+2928.601844333" watchObservedRunningTime="2026-10-30 07:46:04.916374799 +0000 UTC m=+2928.607292240" Oct 30 07:46:04.968175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:04.968103 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" podStartSLOduration=2.9680851969999997 podStartE2EDuration="2.968085197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:04.964691477 +0000 UTC m=+2928.655608916" watchObservedRunningTime="2026-10-30 07:46:04.968085197 +0000 UTC m=+2928.659002636" Oct 30 07:46:05.637799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:05.637761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:07.271425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:07.271384 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:07.397852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:07.396806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:09.118911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:09.118872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:10.828607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:10.828572 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:12.442449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:12.442413 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:14.148444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:14.148410 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:15.820504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:15.820461 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:15.886570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:15.886533 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:46:15.887009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:15.886984 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:46:17.423074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:17.423035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:19.026992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:19.026959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:19.805272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:19.805231 5894 scope.go:122] "RemoveContainer" containerID="d265f31e66c25a0d8eab76c3daf29f668a73d908ab651063b20cb5858f6aeeb8" Oct 30 07:46:19.817039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:19.817011 5894 scope.go:122] "RemoveContainer" containerID="dc89b07b1508192431e0b866aeb7d0319cf850f4a19af057ba8dc4d5f7d4959d" Oct 30 07:46:20.618909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:20.618869 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:22.034509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:22.034476 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx"] Oct 30 07:46:22.038693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:22.038658 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b11c46e-pcnqx"] Oct 30 07:46:22.311385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:22.311292 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:22.433831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:22.433798 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46bc9014-baee-420d-a6d5-df5ffcec87e7" path="/var/lib/kubelet/pods/46bc9014-baee-420d-a6d5-df5ffcec87e7/volumes" Oct 30 07:46:23.946526 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:23.946483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:25.623793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:25.623756 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:27.386001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:27.385959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:29.031591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:29.031550 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn"] Oct 30 07:46:29.043322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:29.043293 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-876ec31b-x68sn"] Oct 30 07:46:29.187505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:29.187469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:30.434414 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.434383 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda4c275-529d-424e-8b75-ac6e910437fb" path="/var/lib/kubelet/pods/cda4c275-529d-424e-8b75-ac6e910437fb/volumes" Oct 30 07:46:30.674013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.673974 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:30.786446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.786413 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:30.903047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.903014 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:30.977285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.977247 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:46:30.986129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:30.986096 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.004529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.004497 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:46:31.008768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.008743 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-e1f1c7\"/\"openshift-service-ca.crt\"" Oct 30 07:46:31.014155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.014133 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-e1f1c7\"/\"e2e-usage-sim-99fb7f-cert\"" Oct 30 07:46:31.014348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.014170 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-e1f1c7\"/\"default-dockercfg-998t5\"" Oct 30 07:46:31.014348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.014275 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-e1f1c7\"/\"kube-root-ca.crt\"" Oct 30 07:46:31.136712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.117340 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.136712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.117444 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26sd\" (UniqueName: \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.226589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.226551 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.226941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.226916 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26sd\" (UniqueName: \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.237025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.236915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:31.255616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.255566 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.294484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.294411 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26sd\" (UniqueName: \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\") pod \"e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.317673 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.313147 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:31.380937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.376061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:31.518711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.517746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:31.718460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.718430 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:31.856976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:31.856600 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:46:32.268713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.268066 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" event={"ID":"d01bda45-a30e-467e-bf44-126e88e59003","Type":"ContainerStarted","Data":"5a72d8cb59114a90d91bd8cac392f692af9e355fba21ae3e2564df134e4639df"} Oct 30 07:46:32.268713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.268119 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" event={"ID":"d01bda45-a30e-467e-bf44-126e88e59003","Type":"ContainerStarted","Data":"a92346721a25b414ccf85269623002c372972b2d5b8a3c4e2494c1be3ce5e16d"} Oct 30 07:46:32.270395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.269345 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:32.270395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.269925 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" podUID="d01bda45-a30e-467e-bf44-126e88e59003" containerName="sim" probeResult="failure" output="Get \"https://10.134.1.52:8000/ready\": dial tcp 10.134.1.52:8000: connect: connection refused" Oct 30 07:46:32.274603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.274561 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:32.384912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.378339 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" podStartSLOduration=2.378318963 podStartE2EDuration="2.378318963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:32.376049842 +0000 UTC m=+2956.066967280" watchObservedRunningTime="2026-10-30 07:46:32.378318963 +0000 UTC m=+2956.069236401" Oct 30 07:46:32.979660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:32.975495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:33.206455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.206417 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:46:33.212892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.212866 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.231600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.231534 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:46:33.238429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.238398 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-b3a7d7\"/\"agc-cred-427e4bf7663d\"" Oct 30 07:46:33.279432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.279400 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:46:33.393883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.393843 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h785\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.394093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.394004 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.394165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.394104 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.394165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.394146 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.495570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.495470 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h785\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.495794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.495575 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.495794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.495630 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.495794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.495719 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.496381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.496353 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.496504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.496460 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.498260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.498237 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.519096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.519070 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h785\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.536892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.536856 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:33.721856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.721823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:33.856222 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:33.856187 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe6447b_aba6_44f4_8ee3_a1dcfb460dd7.slice/crio-e48a1dc0609c4ccb808027ff7183e53d0876ad71f15d4fc3666fc11a4b15d582 WatchSource:0}: Error finding container e48a1dc0609c4ccb808027ff7183e53d0876ad71f15d4fc3666fc11a4b15d582: Status 404 returned error can't find the container with id e48a1dc0609c4ccb808027ff7183e53d0876ad71f15d4fc3666fc11a4b15d582 Oct 30 07:46:33.857570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:33.857542 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:46:34.124768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.124679 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:46:34.130696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.130612 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.148377 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.148347 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-a91a83\"/\"agc-cred-75f95e296368\"" Oct 30 07:46:34.149492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.149455 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:46:34.288477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.288436 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" event={"ID":"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7","Type":"ContainerStarted","Data":"ec148d7fcef96b4f5fbab260125551a9e98a96653897994ad905edc55b26aad3"} Oct 30 07:46:34.288734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.288504 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" event={"ID":"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7","Type":"ContainerStarted","Data":"e48a1dc0609c4ccb808027ff7183e53d0876ad71f15d4fc3666fc11a4b15d582"} Oct 30 07:46:34.288734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.288576 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:34.309030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.308998 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.309212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.309191 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qjd\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.309324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.309306 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.309454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.309372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.334243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.334111 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" podStartSLOduration=1.334081338 podStartE2EDuration="1.334081338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:34.330172251 +0000 UTC m=+2958.021089688" watchObservedRunningTime="2026-10-30 07:46:34.334081338 +0000 UTC m=+2958.024998778" Oct 30 07:46:34.410992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.410907 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qjd\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.410992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.410989 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.411202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.411174 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.411337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.411317 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.411880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.411854 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.412005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.411906 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.414062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.414034 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.434444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.434413 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qjd\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.455175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.455144 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:34.764757 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:34.764706 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8446c48_b973_4fc9_afa5_a2d42a157412.slice/crio-92f4edde77f21289027523b51a2ee890c10878ec48f6ef55fc4b3da148135b82 WatchSource:0}: Error finding container 92f4edde77f21289027523b51a2ee890c10878ec48f6ef55fc4b3da148135b82: Status 404 returned error can't find the container with id 92f4edde77f21289027523b51a2ee890c10878ec48f6ef55fc4b3da148135b82 Oct 30 07:46:34.766333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:34.766306 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:46:34.842055 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:34.842019 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:46:35.033352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.033311 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq"] Oct 30 07:46:35.042134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.042093 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b20aac82-s2jhq"] Oct 30 07:46:35.302519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.302407 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" event={"ID":"c8446c48-b973-4fc9-afa5-a2d42a157412","Type":"ContainerStarted","Data":"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002"} Oct 30 07:46:35.302963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.302521 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" event={"ID":"c8446c48-b973-4fc9-afa5-a2d42a157412","Type":"ContainerStarted","Data":"92f4edde77f21289027523b51a2ee890c10878ec48f6ef55fc4b3da148135b82"} Oct 30 07:46:35.303041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.303019 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:35.344425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.344374 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" podStartSLOduration=1.344360084 podStartE2EDuration="1.344360084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:35.343783302 +0000 UTC m=+2959.034700739" watchObservedRunningTime="2026-10-30 07:46:35.344360084 +0000 UTC m=+2959.035277521" Oct 30 07:46:35.397988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:35.397943 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:36.437644 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:36.437605 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="862dfeb2-98bb-4b60-bbda-bb33805957bb" path="/var/lib/kubelet/pods/862dfeb2-98bb-4b60-bbda-bb33805957bb/volumes" Oct 30 07:46:36.439892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:36.439843 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:36.697548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:36.697467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:37.004498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:37.004398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:37.136161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:37.136123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:37.599412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:37.599380 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:38.420875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.420838 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:46:38.428565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.428531 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.448345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.448310 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:46:38.449988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.449928 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-705337\"/\"agc-cred-8f972a85135b\"" Oct 30 07:46:38.500523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.500487 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.500709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.500558 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.501069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.500814 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmk64\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.501069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.500966 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.602404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.602367 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.602950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.602457 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.602950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.602673 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmk64\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.602950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.602735 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.603203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.603178 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.603554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.603528 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.605272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.605245 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.627101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.627071 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmk64\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\") pod \"payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:38.758117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:38.758031 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:39.072099 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:39.072064 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cb558c_6d46_40cb_b810_7c56f87dc3e1.slice/crio-3a923966f4c0b3bb9bf6b93bd5b3e9907973cbb3fa0cc74229eabf3c078dacd2 WatchSource:0}: Error finding container 3a923966f4c0b3bb9bf6b93bd5b3e9907973cbb3fa0cc74229eabf3c078dacd2: Status 404 returned error can't find the container with id 3a923966f4c0b3bb9bf6b93bd5b3e9907973cbb3fa0cc74229eabf3c078dacd2 Oct 30 07:46:39.074288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.074255 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:46:39.344778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.344695 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:46:39.349403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.349380 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.359369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.359339 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" event={"ID":"d4cb558c-6d46-40cb-b810-7c56f87dc3e1","Type":"ContainerStarted","Data":"4a64138a514e4cd18add049e6f006ef183ea1690ed4f9a451eb26576ca00a3dd"} Oct 30 07:46:39.359521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.359405 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:39.359521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.359428 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" event={"ID":"d4cb558c-6d46-40cb-b810-7c56f87dc3e1","Type":"ContainerStarted","Data":"3a923966f4c0b3bb9bf6b93bd5b3e9907973cbb3fa0cc74229eabf3c078dacd2"} Oct 30 07:46:39.365313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.365288 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bffe43\"/\"agc-cred-e5bf70ebc13b\"" Oct 30 07:46:39.370592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.370566 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:46:39.421398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.421349 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.421622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.421440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68k7l\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.421622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.421475 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.421769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.421720 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.437021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.436965 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" podStartSLOduration=1.436950946 podStartE2EDuration="1.436950946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:39.434039135 +0000 UTC m=+2963.124956573" watchObservedRunningTime="2026-10-30 07:46:39.436950946 +0000 UTC m=+2963.127868382" Oct 30 07:46:39.523219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.523175 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.523443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.523386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.523516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.523441 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68k7l\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.523516 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.523469 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.524032 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.524006 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.524153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.524035 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.526026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.525999 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.547203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.547171 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68k7l\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.666195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.666097 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:39.684995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.684958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:39.985957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:39.985929 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:46:39.986276 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:39.986253 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3c06c6_79cc_459b_990a_3547ee71b5b7.slice/crio-25d4e4b41e62aceac067a4f9fba8b761ddf69182c444c71ad521c283821704d8 WatchSource:0}: Error finding container 25d4e4b41e62aceac067a4f9fba8b761ddf69182c444c71ad521c283821704d8: Status 404 returned error can't find the container with id 25d4e4b41e62aceac067a4f9fba8b761ddf69182c444c71ad521c283821704d8 Oct 30 07:46:40.335840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.335798 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:40.371940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.371905 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" event={"ID":"1f3c06c6-79cc-459b-990a-3547ee71b5b7","Type":"ContainerStarted","Data":"03149e4864cf13d645cb9d168d1952f30ff79fcafc1c5c000c3d3ad62b2230ea"} Oct 30 07:46:40.372129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.371970 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" event={"ID":"1f3c06c6-79cc-459b-990a-3547ee71b5b7","Type":"ContainerStarted","Data":"25d4e4b41e62aceac067a4f9fba8b761ddf69182c444c71ad521c283821704d8"} Oct 30 07:46:40.372129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.371997 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:40.416849 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.416792 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" podStartSLOduration=1.41677554 podStartE2EDuration="1.41677554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:40.41384634 +0000 UTC m=+2964.104763776" watchObservedRunningTime="2026-10-30 07:46:40.41677554 +0000 UTC m=+2964.107693005" Oct 30 07:46:40.455733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.455675 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:40.512895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.512861 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:46:40.517947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.517927 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.536965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.536934 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:46:40.542732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.542610 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-61f6c2-dockercfg-9gbnd\"" Oct 30 07:46:40.542732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.542610 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-61f6c2\"" Oct 30 07:46:40.643718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.642874 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9gc9\" (UniqueName: \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.643718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.642944 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.749161 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.749122 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9gc9\" (UniqueName: \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.750224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.750159 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.751629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.751601 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.781008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.780941 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9gc9\" (UniqueName: \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\") pod \"payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:40.835862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:40.835830 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:41.157125 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:46:41.157039 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e594df_ee57_4057_872a_6db795d02a0e.slice/crio-09e4956dbe8428c345ef49751be2d31d666493ca47601a9ab2dd83d0cdfad7e3 WatchSource:0}: Error finding container 09e4956dbe8428c345ef49751be2d31d666493ca47601a9ab2dd83d0cdfad7e3: Status 404 returned error can't find the container with id 09e4956dbe8428c345ef49751be2d31d666493ca47601a9ab2dd83d0cdfad7e3 Oct 30 07:46:41.158314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.158289 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:46:41.389374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.389332 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" event={"ID":"56e594df-ee57-4057-872a-6db795d02a0e","Type":"ContainerStarted","Data":"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab"} Oct 30 07:46:41.389584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.389400 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" event={"ID":"56e594df-ee57-4057-872a-6db795d02a0e","Type":"ContainerStarted","Data":"09e4956dbe8428c345ef49751be2d31d666493ca47601a9ab2dd83d0cdfad7e3"} Oct 30 07:46:41.389691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.389670 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:41.434166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.434048 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" podStartSLOduration=1.43403446 podStartE2EDuration="1.43403446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:46:41.430520935 +0000 UTC m=+2965.121438375" watchObservedRunningTime="2026-10-30 07:46:41.43403446 +0000 UTC m=+2965.124951896" Oct 30 07:46:41.552445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:41.552390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:43.241226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:43.241191 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:44.992039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:44.991995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:45.306797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:45.306762 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:46:45.386555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:45.386516 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:46:46.315092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:46.315060 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:46:46.405310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:46.405269 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:46:46.773068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:46.772970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:48.538012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:48.537969 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:50.236077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.236038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:50.376341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.376312 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:46:50.387149 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.387118 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" podUID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" containerName="payload-processing" containerID="cri-o://5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3" gracePeriod=25 Oct 30 07:46:50.469092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.469057 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:46:50.679232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.679203 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:46:50.728131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.728093 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\") pod \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " Oct 30 07:46:50.728352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.728213 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\") pod \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " Oct 30 07:46:50.728602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.728552 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\") pod \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " Oct 30 07:46:50.728945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.728725 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume" pod "7d04a175-f5c0-4850-8265-bfc4acfafb5b" (UID: "7d04a175-f5c0-4850-8265-bfc4acfafb5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:50.729162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.729077 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume" pod "7d04a175-f5c0-4850-8265-bfc4acfafb5b" (UID: "7d04a175-f5c0-4850-8265-bfc4acfafb5b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:50.729375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.729350 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\") pod \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\" (UID: \"7d04a175-f5c0-4850-8265-bfc4acfafb5b\") " Oct 30 07:46:50.730126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.730103 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:50.730261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.730128 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7d04a175-f5c0-4850-8265-bfc4acfafb5b-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:50.731118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.731083 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv" pod "7d04a175-f5c0-4850-8265-bfc4acfafb5b" (UID: "7d04a175-f5c0-4850-8265-bfc4acfafb5b"). InnerVolumeSpecName "kube-api-access-897kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:50.731767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.731737 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials" pod "7d04a175-f5c0-4850-8265-bfc4acfafb5b" (UID: "7d04a175-f5c0-4850-8265-bfc4acfafb5b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:50.833189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.833152 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-897kv\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-kube-api-access-897kv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:50.833491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:50.833467 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7d04a175-f5c0-4850-8265-bfc4acfafb5b-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:51.393338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.393302 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:46:51.405909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.405866 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" podUID="193aef98-5299-4ada-90bc-e33852b5562c" containerName="payload-processing" containerID="cri-o://edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965" gracePeriod=25 Oct 30 07:46:51.461259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.461220 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:51.475121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.475083 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:46:51.489525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.489485 5894 generic.go:356] "Generic (PLEG): container finished" podID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" containerID="5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3" exitCode=0 Oct 30 07:46:51.489738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.489602 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" Oct 30 07:46:51.489845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.489824 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" event={"ID":"7d04a175-f5c0-4850-8265-bfc4acfafb5b","Type":"ContainerDied","Data":"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3"} Oct 30 07:46:51.489900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.489882 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm" event={"ID":"7d04a175-f5c0-4850-8265-bfc4acfafb5b","Type":"ContainerDied","Data":"92f77ebcb8b3c08b3cf3e376acf8a244503763cad1590ab9acfa024dfcfb9179"} Oct 30 07:46:51.489939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.489918 5894 scope.go:122] "RemoveContainer" containerID="5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3" Oct 30 07:46:51.521342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.521314 5894 scope.go:122] "RemoveContainer" containerID="5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3" Oct 30 07:46:51.521776 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:51.521753 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3\": container with ID starting with 5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3 not found: ID does not exist" containerID="5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3" Oct 30 07:46:51.521896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.521790 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3"} err="failed to get container status \"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3\": rpc error: code = NotFound desc = could not find container \"5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3\": container with ID starting with 5b0b55a4e0b6f1c5a8a2de43c6563f5c36a1f57a7d6fae4c651b0837828c58c3 not found: ID does not exist" Oct 30 07:46:51.559712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.559679 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:46:51.564540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.564494 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d6fctm"] Oct 30 07:46:51.727506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.727475 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:46:51.849920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.849880 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\") pod \"193aef98-5299-4ada-90bc-e33852b5562c\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " Oct 30 07:46:51.850108 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.849994 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\") pod \"193aef98-5299-4ada-90bc-e33852b5562c\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " Oct 30 07:46:51.850207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850131 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\") pod \"193aef98-5299-4ada-90bc-e33852b5562c\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " Oct 30 07:46:51.850207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850182 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\") pod \"193aef98-5299-4ada-90bc-e33852b5562c\" (UID: \"193aef98-5299-4ada-90bc-e33852b5562c\") " Oct 30 07:46:51.850323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850266 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume" pod "193aef98-5299-4ada-90bc-e33852b5562c" (UID: "193aef98-5299-4ada-90bc-e33852b5562c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:51.850451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850428 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume" pod "193aef98-5299-4ada-90bc-e33852b5562c" (UID: "193aef98-5299-4ada-90bc-e33852b5562c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:51.850861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850843 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:51.850861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.850861 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/193aef98-5299-4ada-90bc-e33852b5562c-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:51.852506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.852475 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr" pod "193aef98-5299-4ada-90bc-e33852b5562c" (UID: "193aef98-5299-4ada-90bc-e33852b5562c"). InnerVolumeSpecName "kube-api-access-z99cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:51.852618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.852519 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials" pod "193aef98-5299-4ada-90bc-e33852b5562c" (UID: "193aef98-5299-4ada-90bc-e33852b5562c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:51.952783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.952693 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99cr\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-kube-api-access-z99cr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:51.952993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:51.952972 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/193aef98-5299-4ada-90bc-e33852b5562c-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:52.000659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.000601 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:52.128658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.125507 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:52.401603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.401571 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:46:52.434889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.434747 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" path="/var/lib/kubelet/pods/7d04a175-f5c0-4850-8265-bfc4acfafb5b/volumes" Oct 30 07:46:52.503806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.503750 5894 generic.go:356] "Generic (PLEG): container finished" podID="193aef98-5299-4ada-90bc-e33852b5562c" containerID="edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965" exitCode=0 Oct 30 07:46:52.503806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.503800 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" event={"ID":"193aef98-5299-4ada-90bc-e33852b5562c","Type":"ContainerDied","Data":"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965"} Oct 30 07:46:52.504038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.503832 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" Oct 30 07:46:52.504038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.503852 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn" event={"ID":"193aef98-5299-4ada-90bc-e33852b5562c","Type":"ContainerDied","Data":"5974fd6295d415707eddbd32f1c08fc0b82073cde33d136dd0288e4635653ecf"} Oct 30 07:46:52.504038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.503884 5894 scope.go:122] "RemoveContainer" containerID="edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965" Oct 30 07:46:52.516063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.516038 5894 scope.go:122] "RemoveContainer" containerID="edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965" Oct 30 07:46:52.516418 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:52.516387 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965\": container with ID starting with edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965 not found: ID does not exist" containerID="edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965" Oct 30 07:46:52.516598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.516428 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965"} err="failed to get container status \"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965\": rpc error: code = NotFound desc = could not find container \"edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965\": container with ID starting with edcd34cc998206ac85e9b43bf78c44adac508fd602797ed7df35bd4930b5c965 not found: ID does not exist" Oct 30 07:46:52.566860 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.566828 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:46:52.571510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:52.571482 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c979jmn"] Oct 30 07:46:53.843076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:53.843037 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:54.433443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:54.433411 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="193aef98-5299-4ada-90bc-e33852b5562c" path="/var/lib/kubelet/pods/193aef98-5299-4ada-90bc-e33852b5562c/volumes" Oct 30 07:46:55.426085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.426046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:55.469556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.469505 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" podUID="e914bca0-18ed-478b-a320-bfa8272bc806" containerName="payload-processing" containerID="cri-o://aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c" gracePeriod=25 Oct 30 07:46:55.768055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.768025 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:46:55.816317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.816280 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\") pod \"e914bca0-18ed-478b-a320-bfa8272bc806\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " Oct 30 07:46:55.816523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.816391 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\") pod \"e914bca0-18ed-478b-a320-bfa8272bc806\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " Oct 30 07:46:55.816523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.816431 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\") pod \"e914bca0-18ed-478b-a320-bfa8272bc806\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " Oct 30 07:46:55.816523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.816517 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\") pod \"e914bca0-18ed-478b-a320-bfa8272bc806\" (UID: \"e914bca0-18ed-478b-a320-bfa8272bc806\") " Oct 30 07:46:55.817758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.816922 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume" pod "e914bca0-18ed-478b-a320-bfa8272bc806" (UID: "e914bca0-18ed-478b-a320-bfa8272bc806"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:55.817758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.817229 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume" pod "e914bca0-18ed-478b-a320-bfa8272bc806" (UID: "e914bca0-18ed-478b-a320-bfa8272bc806"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:55.817758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.817710 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:55.817758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.817734 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e914bca0-18ed-478b-a320-bfa8272bc806-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:55.820555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.820062 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7" pod "e914bca0-18ed-478b-a320-bfa8272bc806" (UID: "e914bca0-18ed-478b-a320-bfa8272bc806"). InnerVolumeSpecName "kube-api-access-p2gt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:55.822669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.822598 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials" pod "e914bca0-18ed-478b-a320-bfa8272bc806" (UID: "e914bca0-18ed-478b-a320-bfa8272bc806"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:55.923406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.920281 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:55.923406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:55.920340 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2gt7\" (UniqueName: \"kubernetes.io/projected/e914bca0-18ed-478b-a320-bfa8272bc806-kube-api-access-p2gt7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:56.475537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.475483 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" podUID="321c6cff-1a60-4d16-a79f-dc322478a6de" containerName="payload-processing" containerID="cri-o://d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe" gracePeriod=25 Oct 30 07:46:56.548926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.548889 5894 generic.go:356] "Generic (PLEG): container finished" podID="e914bca0-18ed-478b-a320-bfa8272bc806" containerID="aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c" exitCode=0 Oct 30 07:46:56.549125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.548948 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" Oct 30 07:46:56.549125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.548971 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" event={"ID":"e914bca0-18ed-478b-a320-bfa8272bc806","Type":"ContainerDied","Data":"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c"} Oct 30 07:46:56.549125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.549037 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x" event={"ID":"e914bca0-18ed-478b-a320-bfa8272bc806","Type":"ContainerDied","Data":"4b8c786d40be6a0579069a332e64fa7c34317c587b432a181336117742c7dce1"} Oct 30 07:46:56.549125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.549067 5894 scope.go:122] "RemoveContainer" containerID="aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c" Oct 30 07:46:56.589306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.589278 5894 scope.go:122] "RemoveContainer" containerID="aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c" Oct 30 07:46:56.589614 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:56.589591 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c\": container with ID starting with aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c not found: ID does not exist" containerID="aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c" Oct 30 07:46:56.589722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.589621 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c"} err="failed to get container status \"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c\": rpc error: code = NotFound desc = could not find container \"aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c\": container with ID starting with aa72c6251d866c89594bbe5b0e4dc308130ef8580b4f8cb5731f24ec3e4f7b3c not found: ID does not exist" Oct 30 07:46:56.603800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.603764 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:46:56.616278 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.616249 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759cct2x"] Oct 30 07:46:56.758890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.758861 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:46:56.834722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.834682 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\") pod \"321c6cff-1a60-4d16-a79f-dc322478a6de\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " Oct 30 07:46:56.834945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.834866 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\") pod \"321c6cff-1a60-4d16-a79f-dc322478a6de\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " Oct 30 07:46:56.834945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.834933 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\") pod \"321c6cff-1a60-4d16-a79f-dc322478a6de\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " Oct 30 07:46:56.835089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.835026 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\") pod \"321c6cff-1a60-4d16-a79f-dc322478a6de\" (UID: \"321c6cff-1a60-4d16-a79f-dc322478a6de\") " Oct 30 07:46:56.835467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.835438 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume" pod "321c6cff-1a60-4d16-a79f-dc322478a6de" (UID: "321c6cff-1a60-4d16-a79f-dc322478a6de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:56.835610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.835488 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume" pod "321c6cff-1a60-4d16-a79f-dc322478a6de" (UID: "321c6cff-1a60-4d16-a79f-dc322478a6de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:46:56.835954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.835891 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:56.835954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.835921 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/321c6cff-1a60-4d16-a79f-dc322478a6de-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:56.838015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.837991 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8" pod "321c6cff-1a60-4d16-a79f-dc322478a6de" (UID: "321c6cff-1a60-4d16-a79f-dc322478a6de"). InnerVolumeSpecName "kube-api-access-zvjn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:56.838122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.838032 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials" pod "321c6cff-1a60-4d16-a79f-dc322478a6de" (UID: "321c6cff-1a60-4d16-a79f-dc322478a6de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:46:56.937128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.937086 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvjn8\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-kube-api-access-zvjn8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:56.937128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:56.937119 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/321c6cff-1a60-4d16-a79f-dc322478a6de-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:46:57.127979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.127945 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:46:57.559107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.559068 5894 generic.go:356] "Generic (PLEG): container finished" podID="321c6cff-1a60-4d16-a79f-dc322478a6de" containerID="d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe" exitCode=0 Oct 30 07:46:57.559507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.559119 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" Oct 30 07:46:57.559507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.559155 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" event={"ID":"321c6cff-1a60-4d16-a79f-dc322478a6de","Type":"ContainerDied","Data":"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe"} Oct 30 07:46:57.559507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.559211 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc" event={"ID":"321c6cff-1a60-4d16-a79f-dc322478a6de","Type":"ContainerDied","Data":"c62e05e95922e31fc934a7623498d47a218a9a55361f31febeae1061fd71f40e"} Oct 30 07:46:57.559507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.559239 5894 scope.go:122] "RemoveContainer" containerID="d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe" Oct 30 07:46:57.570517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.570497 5894 scope.go:122] "RemoveContainer" containerID="d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe" Oct 30 07:46:57.570847 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:46:57.570825 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe\": container with ID starting with d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe not found: ID does not exist" containerID="d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe" Oct 30 07:46:57.570917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.570855 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe"} err="failed to get container status \"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe\": rpc error: code = NotFound desc = could not find container \"d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe\": container with ID starting with d3b74852d0fc8577ae2c165f44e5bb7a8ccb7e99b0092249ac2be76c2b68f2fe not found: ID does not exist" Oct 30 07:46:57.615576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.615541 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:46:57.629042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:57.629005 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfj7rrc"] Oct 30 07:46:58.433973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:58.433937 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321c6cff-1a60-4d16-a79f-dc322478a6de" path="/var/lib/kubelet/pods/321c6cff-1a60-4d16-a79f-dc322478a6de/volumes" Oct 30 07:46:58.434424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:58.434409 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e914bca0-18ed-478b-a320-bfa8272bc806" path="/var/lib/kubelet/pods/e914bca0-18ed-478b-a320-bfa8272bc806/volumes" Oct 30 07:46:59.042786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:46:59.042752 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:00.719373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:00.719332 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:02.341053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:02.341015 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:04.157555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:04.157504 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:04.830482 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:47:04.830442 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:47:04.843911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:04.843880 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:05.327238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:05.327201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:05.816208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:05.816148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:05.950655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:05.949145 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:06.088224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:06.088134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:07.632396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:07.632363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:08.751084 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:08.751050 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:47:09.529654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:09.529596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:10.068155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:10.056503 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:10.183493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:10.183460 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:11.711539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:11.711500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:11.837793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:11.837759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:11.951924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:11.951890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:12.015592 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.015509 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:47:12.016416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016399 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" containerName="payload-processing" Oct 30 07:47:12.016468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016419 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" containerName="payload-processing" Oct 30 07:47:12.016468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016440 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e914bca0-18ed-478b-a320-bfa8272bc806" containerName="payload-processing" Oct 30 07:47:12.016468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016445 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e914bca0-18ed-478b-a320-bfa8272bc806" containerName="payload-processing" Oct 30 07:47:12.016468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016462 5894 cpu_manager.go:414] "containerMap: removing container" podUID="321c6cff-1a60-4d16-a79f-dc322478a6de" containerName="payload-processing" Oct 30 07:47:12.016468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016467 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321c6cff-1a60-4d16-a79f-dc322478a6de" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016485 5894 cpu_manager.go:414] "containerMap: removing container" podUID="193aef98-5299-4ada-90bc-e33852b5562c" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016490 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="193aef98-5299-4ada-90bc-e33852b5562c" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016649 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d04a175-f5c0-4850-8265-bfc4acfafb5b" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016662 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e914bca0-18ed-478b-a320-bfa8272bc806" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016670 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="321c6cff-1a60-4d16-a79f-dc322478a6de" containerName="payload-processing" Oct 30 07:47:12.016682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.016678 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="193aef98-5299-4ada-90bc-e33852b5562c" containerName="payload-processing" Oct 30 07:47:12.021690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.021668 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.039374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.039341 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:47:12.047070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.047028 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-759ae6\"" Oct 30 07:47:12.047268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.047100 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-759ae6-dockercfg-2rggh\"" Oct 30 07:47:12.071762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.071709 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:47:12.089700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.078116 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.095515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.095462 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:47:12.194667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.188330 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.194667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.188490 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t26dj\" (UniqueName: \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.194667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.188799 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q87w\" (UniqueName: \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.194667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.188849 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.295824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.295785 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q87w\" (UniqueName: \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.296040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.295835 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.296040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.295865 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.296040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.295953 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t26dj\" (UniqueName: \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.297655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.297099 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.297655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.297523 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.322527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.322450 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t26dj\" (UniqueName: \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\") pod \"payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.323772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.323745 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q87w\" (UniqueName: \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\") pod \"payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.344398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.344361 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.591255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.591160 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:12.681900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.681848 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:47:12.699883 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:47:12.699107 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcd7b22_7bca_4bb3_b72f_f714e13d02fb.slice/crio-d0eed0c3287e3149b8e4130b785cfa840a13fde153cd9adbbfbaaf5cc55d4f32 WatchSource:0}: Error finding container d0eed0c3287e3149b8e4130b785cfa840a13fde153cd9adbbfbaaf5cc55d4f32: Status 404 returned error can't find the container with id d0eed0c3287e3149b8e4130b785cfa840a13fde153cd9adbbfbaaf5cc55d4f32 Oct 30 07:47:12.845304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.845202 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" event={"ID":"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb","Type":"ContainerStarted","Data":"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e"} Oct 30 07:47:12.845304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.845279 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:12.845304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.845304 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" event={"ID":"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb","Type":"ContainerStarted","Data":"d0eed0c3287e3149b8e4130b785cfa840a13fde153cd9adbbfbaaf5cc55d4f32"} Oct 30 07:47:12.893686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.893603 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" podStartSLOduration=1.893583244 podStartE2EDuration="1.893583244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:12.887870481 +0000 UTC m=+2996.578787919" watchObservedRunningTime="2026-10-30 07:47:12.893583244 +0000 UTC m=+2996.584500682" Oct 30 07:47:12.906777 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:47:12.906739 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb224b8a5_9cf6_4cc1_a379_6dcc1e565cae.slice/crio-a4a0cd52d175da808434569a1d5af1250ac134ec1911754ee58ce2156d7dc00c WatchSource:0}: Error finding container a4a0cd52d175da808434569a1d5af1250ac134ec1911754ee58ce2156d7dc00c: Status 404 returned error can't find the container with id a4a0cd52d175da808434569a1d5af1250ac134ec1911754ee58ce2156d7dc00c Oct 30 07:47:12.907308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:12.907283 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:47:13.764065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:13.764029 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:13.855803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:13.855767 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" event={"ID":"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae","Type":"ContainerStarted","Data":"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97"} Oct 30 07:47:13.856247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:13.855821 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" event={"ID":"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae","Type":"ContainerStarted","Data":"a4a0cd52d175da808434569a1d5af1250ac134ec1911754ee58ce2156d7dc00c"} Oct 30 07:47:13.856247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:13.855859 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:13.900098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:13.900041 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" podStartSLOduration=1.9000260610000002 podStartE2EDuration="1.900026061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:13.896679997 +0000 UTC m=+2997.587597433" watchObservedRunningTime="2026-10-30 07:47:13.900026061 +0000 UTC m=+2997.590943494" Oct 30 07:47:15.410154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:15.410117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:16.033616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:16.033578 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx"] Oct 30 07:47:16.045558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:16.045523 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03c6453b-cphfx"] Oct 30 07:47:16.434219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:16.434134 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d885cdeb-aef2-4231-947a-960e8f40639b" path="/var/lib/kubelet/pods/d885cdeb-aef2-4231-947a-960e8f40639b/volumes" Oct 30 07:47:17.034075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:17.034038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:18.035127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:18.035088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:18.648850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:18.648815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:19.972510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:19.972478 5894 scope.go:122] "RemoveContainer" containerID="aab81015c13238c77bed226c697b148f1db00444c8ab1aca78b63263b6e04095" Oct 30 07:47:19.983704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:19.983678 5894 scope.go:122] "RemoveContainer" containerID="2d4389edb580439e270241c3a22878e3f135c68b537e94da332b2e5842a42869" Oct 30 07:47:20.422158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:20.422125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:22.120796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:22.120757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:23.692903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:23.692865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:23.859830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:23.859798 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:47:24.868848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:24.868817 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:47:25.439332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:25.439293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:27.119946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:27.119908 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:28.034877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:28.034846 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn"] Oct 30 07:47:28.046014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:28.045980 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-843d03e2-flxmn"] Oct 30 07:47:28.433129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:28.433046 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c19ed8-c212-4d43-b14f-80adba57dbcf" path="/var/lib/kubelet/pods/73c19ed8-c212-4d43-b14f-80adba57dbcf/volumes" Oct 30 07:47:28.705408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:28.705327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:28.815821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:28.815780 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:30.640955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:30.640917 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:31.329318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:31.329271 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:32.273351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:32.273314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:33.937821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:33.937779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:34.828774 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:47:34.828733 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:47:35.031033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:35.030995 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl"] Oct 30 07:47:35.043305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:35.043275 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8992c559-n4vfl"] Oct 30 07:47:35.723459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:35.723422 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:36.434135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:36.434098 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574bad66-f295-4324-842f-19f180c7e65d" path="/var/lib/kubelet/pods/574bad66-f295-4324-842f-19f180c7e65d/volumes" Oct 30 07:47:37.351188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:37.351145 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:39.031289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:39.031241 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:40.031417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:40.031373 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd"] Oct 30 07:47:40.043681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:40.043645 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c0b593a-t88rd"] Oct 30 07:47:40.434320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:40.434241 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4251461-cf8f-4484-87c3-fff9d3c3dce6" path="/var/lib/kubelet/pods/f4251461-cf8f-4484-87c3-fff9d3c3dce6/volumes" Oct 30 07:47:40.630914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:40.630879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:42.420145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:42.420115 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:44.021121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:44.021085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:45.675120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:45.675083 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:47.473565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:47.473522 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:48.326077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:48.326043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:49.458863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:49.458824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:50.172819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:50.172775 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:47:51.267074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.267040 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:51.732955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.732911 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:47:51.738492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.738464 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.761320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.761286 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:47:51.768677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.768489 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5f604\"/\"openshift-service-ca.crt\"" Oct 30 07:47:51.768677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.768573 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5f604\"/\"test-model-d5f604-kserve-self-signed-certs\"" Oct 30 07:47:51.768677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.768597 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5f604\"/\"kube-root-ca.crt\"" Oct 30 07:47:51.768980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.768703 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5f604\"/\"default-dockercfg-vd2t5\"" Oct 30 07:47:51.794375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.794335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:51.812819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.812772 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.813004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.812838 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.813004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.812972 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstcf\" (UniqueName: \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.813101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.813025 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.813170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.813153 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.813242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.813227 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914235 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914260 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914293 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstcf\" (UniqueName: \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914322 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914397 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914530 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914621 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.914759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.914741 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.918093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.918067 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.919583 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.919555 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:51.940372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:51.940340 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstcf\" (UniqueName: \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\") pod \"test-model-d5f604-kserve-7855cb9697-zsv6t\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:52.058666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:52.058012 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:52.480094 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:47:52.480061 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88afd2e9_56a1_4217_9d8c_7b3def0982d7.slice/crio-c6bb694ddfc889c88c11f8eae4b99a2369a50344a4b650e7ea76ff84dd043c11 WatchSource:0}: Error finding container c6bb694ddfc889c88c11f8eae4b99a2369a50344a4b650e7ea76ff84dd043c11: Status 404 returned error can't find the container with id c6bb694ddfc889c88c11f8eae4b99a2369a50344a4b650e7ea76ff84dd043c11 Oct 30 07:47:52.480480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:52.480215 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:47:53.142514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:53.142480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:53.267086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:53.267045 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" event={"ID":"88afd2e9-56a1-4217-9d8c-7b3def0982d7","Type":"ContainerStarted","Data":"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be"} Oct 30 07:47:53.267086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:53.267099 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" event={"ID":"88afd2e9-56a1-4217-9d8c-7b3def0982d7","Type":"ContainerStarted","Data":"c6bb694ddfc889c88c11f8eae4b99a2369a50344a4b650e7ea76ff84dd043c11"} Oct 30 07:47:53.310008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:53.309959 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" podStartSLOduration=2.309946145 podStartE2EDuration="2.309946145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:47:53.309239995 +0000 UTC m=+3037.000157432" watchObservedRunningTime="2026-10-30 07:47:53.309946145 +0000 UTC m=+3037.000863582" Oct 30 07:47:54.059561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:54.059526 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:47:54.828141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:54.828108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:56.733282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:56.733245 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:58.036455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:58.036414 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv"] Oct 30 07:47:58.049071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:58.049029 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a64e105e-gcdjv"] Oct 30 07:47:58.392707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:58.392602 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:47:58.467657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:47:58.462591 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cce887b3-2ac0-4009-ae0e-805bd30dc1e4" path="/var/lib/kubelet/pods/cce887b3-2ac0-4009-ae0e-805bd30dc1e4/volumes" Oct 30 07:48:00.071822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:00.071778 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:01.944836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:01.944788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:02.060004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:02.059972 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:48:02.075651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:02.075607 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:48:02.356356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:02.356325 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:48:03.682983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:03.682937 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:03.815535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:03.815482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:04.087890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:04.087850 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:04.848299 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:48:04.848262 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:48:05.454959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:05.454916 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:07.139078 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:07.139031 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:08.884364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:08.876557 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:10.505473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:10.505430 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:12.238266 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:12.238227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:12.981080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:12.981042 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:48:12.987966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:12.987941 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.012520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.009491 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:48:13.015110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.015079 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-63f24b\"/\"default-dockercfg-k4p9f\"" Oct 30 07:48:13.015283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.015183 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-63f24b\"/\"test-model-63f24b-kserve-self-signed-certs\"" Oct 30 07:48:13.015951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.015928 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-63f24b\"/\"kube-root-ca.crt\"" Oct 30 07:48:13.016122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.015928 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-63f24b\"/\"openshift-service-ca.crt\"" Oct 30 07:48:13.069268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.069233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:13.112292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112261 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.112458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112311 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.112525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112485 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5sxh\" (UniqueName: \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.112571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112536 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.112748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112730 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.112834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.112789 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.214671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.214573 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.214909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.214705 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.214909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.214799 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5sxh\" (UniqueName: \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.214909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.214864 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.215097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.215003 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.215097 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.215077 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.215318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.215274 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.215749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.215722 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.215821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.215781 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.217435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.217404 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.218062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.218035 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.279712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.258405 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5sxh\" (UniqueName: \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\") pod \"test-model-63f24b-kserve-6448b94c6f-q4754\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.322664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.321875 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:13.749000 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:13.748969 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bfaf8fe_add8_4dac_83ac_ad7d2fe2d058.slice/crio-0ab57c49c2f98c71766464a4889fb761eca543513d0624fec9420baa6f193e28 WatchSource:0}: Error finding container 0ab57c49c2f98c71766464a4889fb761eca543513d0624fec9420baa6f193e28: Status 404 returned error can't find the container with id 0ab57c49c2f98c71766464a4889fb761eca543513d0624fec9420baa6f193e28 Oct 30 07:48:13.751116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:13.750766 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:48:14.013957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:14.013857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:14.469557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:14.469509 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" event={"ID":"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058","Type":"ContainerStarted","Data":"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882"} Oct 30 07:48:14.470111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:14.469570 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" event={"ID":"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058","Type":"ContainerStarted","Data":"0ab57c49c2f98c71766464a4889fb761eca543513d0624fec9420baa6f193e28"} Oct 30 07:48:14.515071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:14.515019 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" podStartSLOduration=2.515006588 podStartE2EDuration="2.515006588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:14.514739408 +0000 UTC m=+3058.205656845" watchObservedRunningTime="2026-10-30 07:48:14.515006588 +0000 UTC m=+3058.205924024" Oct 30 07:48:15.324142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:15.324106 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:15.830283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:15.830250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:17.558187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:17.558146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:19.208666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.208599 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248"] Oct 30 07:48:19.213392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.213368 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.231965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.231929 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:48:19.234339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.234316 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248"] Oct 30 07:48:19.242235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.242211 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:48:19.242417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.242218 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:48:19.242417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.242224 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:48:19.354549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.354518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:19.363222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.363195 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.363399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.363381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.363593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.363573 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc5j\" (UniqueName: \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.464831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.464746 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.465012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.464930 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.465135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.465116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc5j\" (UniqueName: \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.465529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.465506 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.467382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.467360 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.489281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.489245 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc5j\" (UniqueName: \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\") pod \"maas-api-revoke-sub-da45bd21-sh248\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.530479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.530452 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:19.854402 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:19.854353 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fd062e_b1f8_4c27_8d4e_25e431393aa2.slice/crio-42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa WatchSource:0}: Error finding container 42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa: Status 404 returned error can't find the container with id 42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa Oct 30 07:48:19.856669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.856625 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248"] Oct 30 07:48:19.939565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.939529 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx"] Oct 30 07:48:19.947008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.946981 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:19.963832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:19.963804 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx"] Oct 30 07:48:20.044190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.044155 5894 scope.go:122] "RemoveContainer" containerID="f2ffc4a034d81959b1d890540c4d3d940cdfd4565354c652802ff3991f966a29" Oct 30 07:48:20.055920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.055890 5894 scope.go:122] "RemoveContainer" containerID="7b46a94d71cf2c8f5ea2a4281a0482426fd0fd739a5b8a340fb719efdd855ccf" Oct 30 07:48:20.067063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.067028 5894 scope.go:122] "RemoveContainer" containerID="8ac907ac5589759f235bc1d75bbf0fd63a78cbf334b7f8f5c4a29e888327854b" Oct 30 07:48:20.088918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.088889 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.089085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.089056 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8fb\" (UniqueName: \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.089209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.089111 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.190341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.190297 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.190534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.190432 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8fb\" (UniqueName: \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.190534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.190476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.190964 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.190943 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.192902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.192879 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.214194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.214161 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8fb\" (UniqueName: \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\") pod \"maas-api-revoke-sub-f68e69d7-tpvjx\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.264140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.264103 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:20.544256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.544219 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" event={"ID":"c7fd062e-b1f8-4c27-8d4e-25e431393aa2","Type":"ContainerStarted","Data":"e6c886f05fca0bbae76215dda4f397a0631ad9fe5c684bf89f91e2159cf4c2ac"} Oct 30 07:48:20.544460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.544276 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" event={"ID":"c7fd062e-b1f8-4c27-8d4e-25e431393aa2","Type":"ContainerStarted","Data":"42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa"} Oct 30 07:48:20.571755 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:20.571715 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3521a0ac_248a_492f_bfc4_26d5a624f260.slice/crio-233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949 WatchSource:0}: Error finding container 233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949: Status 404 returned error can't find the container with id 233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949 Oct 30 07:48:20.573606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.573577 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx"] Oct 30 07:48:20.586788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:20.586731 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" podStartSLOduration=1.586711792 podStartE2EDuration="1.586711792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:20.584966984 +0000 UTC m=+3064.275884422" watchObservedRunningTime="2026-10-30 07:48:20.586711792 +0000 UTC m=+3064.277629237" Oct 30 07:48:21.044204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.044170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:21.555246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.555205 5894 generic.go:356] "Generic (PLEG): container finished" podID="c7fd062e-b1f8-4c27-8d4e-25e431393aa2" containerID="e6c886f05fca0bbae76215dda4f397a0631ad9fe5c684bf89f91e2159cf4c2ac" exitCode=0 Oct 30 07:48:21.555730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.555299 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" event={"ID":"c7fd062e-b1f8-4c27-8d4e-25e431393aa2","Type":"ContainerDied","Data":"e6c886f05fca0bbae76215dda4f397a0631ad9fe5c684bf89f91e2159cf4c2ac"} Oct 30 07:48:21.557074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.557049 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" event={"ID":"3521a0ac-248a-492f-bfc4-26d5a624f260","Type":"ContainerStarted","Data":"8d80fb0360f85515d997cccccf3eebf8b0830dab00719f3982736e6547171337"} Oct 30 07:48:21.557219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.557086 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" event={"ID":"3521a0ac-248a-492f-bfc4-26d5a624f260","Type":"ContainerStarted","Data":"233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949"} Oct 30 07:48:21.640525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:21.640469 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" podStartSLOduration=2.640454874 podStartE2EDuration="2.640454874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:21.63775322 +0000 UTC m=+3065.328670659" watchObservedRunningTime="2026-10-30 07:48:21.640454874 +0000 UTC m=+3065.331372310" Oct 30 07:48:22.566771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.566735 5894 generic.go:356] "Generic (PLEG): container finished" podID="3521a0ac-248a-492f-bfc4-26d5a624f260" containerID="8d80fb0360f85515d997cccccf3eebf8b0830dab00719f3982736e6547171337" exitCode=0 Oct 30 07:48:22.567216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.566819 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" event={"ID":"3521a0ac-248a-492f-bfc4-26d5a624f260","Type":"ContainerDied","Data":"8d80fb0360f85515d997cccccf3eebf8b0830dab00719f3982736e6547171337"} Oct 30 07:48:22.747329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.747301 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:22.791529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.790301 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:22.834727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.832250 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\") pod \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " Oct 30 07:48:22.834727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.832361 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\" (UniqueName: \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\") pod \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " Oct 30 07:48:22.834727 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.832611 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\") pod \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\" (UID: \"c7fd062e-b1f8-4c27-8d4e-25e431393aa2\") " Oct 30 07:48:22.838567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.838529 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca" pod "c7fd062e-b1f8-4c27-8d4e-25e431393aa2" (UID: "c7fd062e-b1f8-4c27-8d4e-25e431393aa2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:22.839831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.839795 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca" pod "c7fd062e-b1f8-4c27-8d4e-25e431393aa2" (UID: "c7fd062e-b1f8-4c27-8d4e-25e431393aa2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:22.849983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.849946 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j" pod "c7fd062e-b1f8-4c27-8d4e-25e431393aa2" (UID: "c7fd062e-b1f8-4c27-8d4e-25e431393aa2"). InnerVolumeSpecName "kube-api-access-hbc5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:22.935507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.935462 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:22.935507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.935499 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc5j\" (UniqueName: \"kubernetes.io/projected/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-kube-api-access-hbc5j\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:22.935507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:22.935511 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fd062e-b1f8-4c27-8d4e-25e431393aa2-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:23.324417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.324385 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:23.337355 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.337326 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:23.577238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.577157 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" Oct 30 07:48:23.577238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.577189 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248" event={"ID":"c7fd062e-b1f8-4c27-8d4e-25e431393aa2","Type":"ContainerDied","Data":"42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa"} Oct 30 07:48:23.577238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.577235 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ef88eb97977d07c94def04aeb62971932dcb6c0714a73684f3b4c660faceaa" Oct 30 07:48:23.590187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.590157 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:48:23.767744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.767716 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:23.848760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.848542 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\") pod \"3521a0ac-248a-492f-bfc4-26d5a624f260\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " Oct 30 07:48:23.848956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.848891 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\") pod \"3521a0ac-248a-492f-bfc4-26d5a624f260\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " Oct 30 07:48:23.849035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.848995 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca" pod "3521a0ac-248a-492f-bfc4-26d5a624f260" (UID: "3521a0ac-248a-492f-bfc4-26d5a624f260"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:23.849102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.849056 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\" (UniqueName: \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\") pod \"3521a0ac-248a-492f-bfc4-26d5a624f260\" (UID: \"3521a0ac-248a-492f-bfc4-26d5a624f260\") " Oct 30 07:48:23.849739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.849708 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:23.851685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.851609 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca" pod "3521a0ac-248a-492f-bfc4-26d5a624f260" (UID: "3521a0ac-248a-492f-bfc4-26d5a624f260"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:23.851811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.851704 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb" pod "3521a0ac-248a-492f-bfc4-26d5a624f260" (UID: "3521a0ac-248a-492f-bfc4-26d5a624f260"). InnerVolumeSpecName "kube-api-access-kf8fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:23.951051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.951018 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3521a0ac-248a-492f-bfc4-26d5a624f260-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:23.951051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:23.951046 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8fb\" (UniqueName: \"kubernetes.io/projected/3521a0ac-248a-492f-bfc4-26d5a624f260-kube-api-access-kf8fb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:24.461700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:24.461668 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:24.587518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:24.587485 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" Oct 30 07:48:24.587518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:24.587507 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx" event={"ID":"3521a0ac-248a-492f-bfc4-26d5a624f260","Type":"ContainerDied","Data":"233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949"} Oct 30 07:48:24.588131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:24.587569 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="233b8371224a728f586333930263eff8a6d796750d08115e65043de56bd66949" Oct 30 07:48:26.199082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:26.199040 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:27.929358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:27.929271 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:29.251223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:29.251177 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:29.625437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:29.625385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:30.067467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.067426 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb"] Oct 30 07:48:30.068395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068373 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3521a0ac-248a-492f-bfc4-26d5a624f260" containerName="revoke-keys" Oct 30 07:48:30.068395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068397 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3521a0ac-248a-492f-bfc4-26d5a624f260" containerName="revoke-keys" Oct 30 07:48:30.068593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068460 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c7fd062e-b1f8-4c27-8d4e-25e431393aa2" containerName="revoke-keys" Oct 30 07:48:30.068593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068467 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fd062e-b1f8-4c27-8d4e-25e431393aa2" containerName="revoke-keys" Oct 30 07:48:30.068737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068612 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3521a0ac-248a-492f-bfc4-26d5a624f260" containerName="revoke-keys" Oct 30 07:48:30.068737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.068647 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fd062e-b1f8-4c27-8d4e-25e431393aa2" containerName="revoke-keys" Oct 30 07:48:30.072905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.072883 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.098817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.092196 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb"] Oct 30 07:48:30.103869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.103836 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:48:30.103869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.103867 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:48:30.137993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.137957 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:48:30.138191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.138174 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:48:30.159817 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.159778 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.160000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.159838 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgww5\" (UniqueName: \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.160000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.159958 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.261613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.261565 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.261613 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.261642 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgww5\" (UniqueName: \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.262209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.261747 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.262432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.262407 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.264081 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.264057 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.286051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.286018 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgww5\" (UniqueName: \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\") pod \"maas-api-revoke-sub-3f756628-2f2rb\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.391307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.391224 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:30.711532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.711487 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb"] Oct 30 07:48:30.851567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.851528 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf"] Oct 30 07:48:30.858397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.858370 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:30.876227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.876196 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf"] Oct 30 07:48:30.973755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.973667 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:30.973913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.973788 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:30.974049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:30.974032 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v2kf\" (UniqueName: \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.075358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.075319 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v2kf\" (UniqueName: \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.075562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.075439 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.075562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.075506 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.076240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.076214 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.078122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.078101 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.099434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.099406 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v2kf\" (UniqueName: \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\") pod \"maas-api-revoke-sub-8b35c51c-kvntf\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.175940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.175897 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:31.458270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.458228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:31.527235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.502479 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf"] Oct 30 07:48:31.580111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.580054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:31.650965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.650921 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" event={"ID":"148b79ae-0821-42cf-b1e6-bf1aed3bff76","Type":"ContainerStarted","Data":"fb80a813ed1a718a510fd456aed24743ab870aaa172511076b2b8998b7452470"} Oct 30 07:48:31.651158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.650988 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" event={"ID":"148b79ae-0821-42cf-b1e6-bf1aed3bff76","Type":"ContainerStarted","Data":"e6cac7e6e55087cc9f07517f18d099123a3a374773cf1e19330683a437198570"} Oct 30 07:48:31.652500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.652469 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" event={"ID":"dea9f335-2f30-4d45-8bc0-6c066a67b947","Type":"ContainerStarted","Data":"71f65a6d3dcc39ce791284bfeae1d49f70735a8cccd35d5b55016a48ad7679e4"} Oct 30 07:48:31.695538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:31.695462 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" podStartSLOduration=1.695440606 podStartE2EDuration="1.695440606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:31.692238114 +0000 UTC m=+3075.383155550" watchObservedRunningTime="2026-10-30 07:48:31.695440606 +0000 UTC m=+3075.386358048" Oct 30 07:48:32.663865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:32.663832 5894 generic.go:356] "Generic (PLEG): container finished" podID="148b79ae-0821-42cf-b1e6-bf1aed3bff76" containerID="fb80a813ed1a718a510fd456aed24743ab870aaa172511076b2b8998b7452470" exitCode=0 Oct 30 07:48:32.664348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:32.663926 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" event={"ID":"148b79ae-0821-42cf-b1e6-bf1aed3bff76","Type":"ContainerDied","Data":"fb80a813ed1a718a510fd456aed24743ab870aaa172511076b2b8998b7452470"} Oct 30 07:48:32.665844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:32.665817 5894 generic.go:356] "Generic (PLEG): container finished" podID="dea9f335-2f30-4d45-8bc0-6c066a67b947" containerID="e1ecd98229046d5714504ec34ab721e5a1df9cf3a8dd958d803a866b6f65f56d" exitCode=0 Oct 30 07:48:32.665976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:32.665898 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" event={"ID":"dea9f335-2f30-4d45-8bc0-6c066a67b947","Type":"ContainerDied","Data":"e1ecd98229046d5714504ec34ab721e5a1df9cf3a8dd958d803a866b6f65f56d"} Oct 30 07:48:33.061438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.061402 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:48:33.066912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.066884 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.068985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.068956 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:33.087679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.086580 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fae1d2\"/\"openshift-service-ca.crt\"" Oct 30 07:48:33.087679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.086590 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fae1d2\"/\"kube-root-ca.crt\"" Oct 30 07:48:33.087679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.086695 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fae1d2\"/\"default-dockercfg-8cxz8\"" Oct 30 07:48:33.087679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.086709 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fae1d2\"/\"test-model-fae1d2-kserve-self-signed-certs\"" Oct 30 07:48:33.088308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.088287 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:48:33.197877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.197839 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:33.213361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213324 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.213540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213482 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.213725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213706 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.213815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213741 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844g7\" (UniqueName: \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.213906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213883 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.213995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.213983 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.316740 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.316926 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.317015 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.317178 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.317204 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844g7\" (UniqueName: \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.317758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.317266 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.318218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.317966 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.324667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.318757 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.324667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.319045 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.332455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.332412 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.335840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.335776 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.401660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.397857 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844g7\" (UniqueName: \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\") pod \"test-model-fae1d2-kserve-8ffdf4965-ftlfh\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.687079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.687038 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:33.892404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.892375 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:33.920178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:33.920147 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:34.026702 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:34.026671 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7769b684_807f_49b6_9231_475bc872e040.slice/crio-27cde8c3c513b1d22838822a0de5e972a7c6845f3f35cacf0b1b02da56145a3c WatchSource:0}: Error finding container 27cde8c3c513b1d22838822a0de5e972a7c6845f3f35cacf0b1b02da56145a3c: Status 404 returned error can't find the container with id 27cde8c3c513b1d22838822a0de5e972a7c6845f3f35cacf0b1b02da56145a3c Oct 30 07:48:34.029116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.029073 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:48:34.043379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043348 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\") pod \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " Oct 30 07:48:34.043512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043450 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\") pod \"dea9f335-2f30-4d45-8bc0-6c066a67b947\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " Oct 30 07:48:34.043512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043484 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\" (UniqueName: \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\") pod \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " Oct 30 07:48:34.043656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043558 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\") pod \"dea9f335-2f30-4d45-8bc0-6c066a67b947\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " Oct 30 07:48:34.043726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043711 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\" (UniqueName: \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\") pod \"dea9f335-2f30-4d45-8bc0-6c066a67b947\" (UID: \"dea9f335-2f30-4d45-8bc0-6c066a67b947\") " Oct 30 07:48:34.043900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043873 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca" pod "dea9f335-2f30-4d45-8bc0-6c066a67b947" (UID: "dea9f335-2f30-4d45-8bc0-6c066a67b947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:34.044011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.043985 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\") pod \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\" (UID: \"148b79ae-0821-42cf-b1e6-bf1aed3bff76\") " Oct 30 07:48:34.044561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.044534 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca" pod "148b79ae-0821-42cf-b1e6-bf1aed3bff76" (UID: "148b79ae-0821-42cf-b1e6-bf1aed3bff76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:34.044767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.044741 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.044862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.044771 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.046226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.046198 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf" pod "dea9f335-2f30-4d45-8bc0-6c066a67b947" (UID: "dea9f335-2f30-4d45-8bc0-6c066a67b947"). InnerVolumeSpecName "kube-api-access-5v2kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:34.046568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.046546 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca" pod "148b79ae-0821-42cf-b1e6-bf1aed3bff76" (UID: "148b79ae-0821-42cf-b1e6-bf1aed3bff76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:34.046687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.046574 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5" pod "148b79ae-0821-42cf-b1e6-bf1aed3bff76" (UID: "148b79ae-0821-42cf-b1e6-bf1aed3bff76"). InnerVolumeSpecName "kube-api-access-zgww5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:34.047294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.047244 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca" pod "dea9f335-2f30-4d45-8bc0-6c066a67b947" (UID: "dea9f335-2f30-4d45-8bc0-6c066a67b947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:34.145909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.145864 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgww5\" (UniqueName: \"kubernetes.io/projected/148b79ae-0821-42cf-b1e6-bf1aed3bff76-kube-api-access-zgww5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.145909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.145905 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea9f335-2f30-4d45-8bc0-6c066a67b947-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.146517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.145920 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v2kf\" (UniqueName: \"kubernetes.io/projected/dea9f335-2f30-4d45-8bc0-6c066a67b947-kube-api-access-5v2kf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.146517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.145930 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/148b79ae-0821-42cf-b1e6-bf1aed3bff76-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:34.686195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.686157 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" Oct 30 07:48:34.686195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.686191 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb" event={"ID":"148b79ae-0821-42cf-b1e6-bf1aed3bff76","Type":"ContainerDied","Data":"e6cac7e6e55087cc9f07517f18d099123a3a374773cf1e19330683a437198570"} Oct 30 07:48:34.686548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.686242 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6cac7e6e55087cc9f07517f18d099123a3a374773cf1e19330683a437198570" Oct 30 07:48:34.688014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.687982 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" event={"ID":"7769b684-807f-49b6-9231-475bc872e040","Type":"ContainerStarted","Data":"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68"} Oct 30 07:48:34.688434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.688028 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" event={"ID":"7769b684-807f-49b6-9231-475bc872e040","Type":"ContainerStarted","Data":"27cde8c3c513b1d22838822a0de5e972a7c6845f3f35cacf0b1b02da56145a3c"} Oct 30 07:48:34.689693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.689670 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" Oct 30 07:48:34.689811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.689691 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf" event={"ID":"dea9f335-2f30-4d45-8bc0-6c066a67b947","Type":"ContainerDied","Data":"71f65a6d3dcc39ce791284bfeae1d49f70735a8cccd35d5b55016a48ad7679e4"} Oct 30 07:48:34.689811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.689728 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f65a6d3dcc39ce791284bfeae1d49f70735a8cccd35d5b55016a48ad7679e4" Oct 30 07:48:34.749800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.749745 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" podStartSLOduration=1.749730582 podStartE2EDuration="1.749730582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:34.741069643 +0000 UTC m=+3078.431987080" watchObservedRunningTime="2026-10-30 07:48:34.749730582 +0000 UTC m=+3078.440648019" Oct 30 07:48:34.812488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:34.812446 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:34.832148 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:48:34.832099 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:48:35.687275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:35.687236 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:36.033259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:36.033226 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj"] Oct 30 07:48:36.045437 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:36.045404 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c68a06-l55lj"] Oct 30 07:48:36.402792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:36.402696 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:36.435844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:36.435807 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6289d100-cdda-4627-83fc-5d98ba37e8bf" path="/var/lib/kubelet/pods/6289d100-cdda-4627-83fc-5d98ba37e8bf/volumes" Oct 30 07:48:38.135892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:38.135848 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:39.827203 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:39.827164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:39.964723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:39.962130 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:48:41.790691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:41.790657 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:41.912070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:41.912026 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:42.272283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.272193 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:48:42.273140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273122 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dea9f335-2f30-4d45-8bc0-6c066a67b947" containerName="revoke-keys" Oct 30 07:48:42.273202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273143 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea9f335-2f30-4d45-8bc0-6c066a67b947" containerName="revoke-keys" Oct 30 07:48:42.273202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273186 5894 cpu_manager.go:414] "containerMap: removing container" podUID="148b79ae-0821-42cf-b1e6-bf1aed3bff76" containerName="revoke-keys" Oct 30 07:48:42.273202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273191 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148b79ae-0821-42cf-b1e6-bf1aed3bff76" containerName="revoke-keys" Oct 30 07:48:42.273314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273309 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="148b79ae-0821-42cf-b1e6-bf1aed3bff76" containerName="revoke-keys" Oct 30 07:48:42.273357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.273327 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea9f335-2f30-4d45-8bc0-6c066a67b947" containerName="revoke-keys" Oct 30 07:48:42.278864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.278833 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.297556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.297520 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:48:42.394063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.394020 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xclc\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.394268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.394222 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.394352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.394337 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.394462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.394450 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.435830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.435798 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:48:42.436102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.435966 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" podUID="d01bda45-a30e-467e-bf44-126e88e59003" containerName="sim" containerID="cri-o://5a72d8cb59114a90d91bd8cac392f692af9e355fba21ae3e2564df134e4639df" gracePeriod=30 Oct 30 07:48:42.496261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.496220 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.496429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.496323 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.496429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.496406 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.496569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.496482 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xclc\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.497011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.496984 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.497189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.497166 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.498878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.498861 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.520392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.520354 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xclc\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\") pod \"payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.600665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.599049 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:42.649400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.647560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:42.918394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.912697 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v"] Oct 30 07:48:42.923962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.923832 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v"] Oct 30 07:48:42.923962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.923876 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:42.942008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.940109 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:48:42.942008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.940191 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:48:42.942008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.940381 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:48:42.959596 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.959567 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:48:42.967034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.966997 5894 generic.go:356] "Generic (PLEG): container finished" podID="d01bda45-a30e-467e-bf44-126e88e59003" containerID="5a72d8cb59114a90d91bd8cac392f692af9e355fba21ae3e2564df134e4639df" exitCode=0 Oct 30 07:48:42.967738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.967703 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" event={"ID":"d01bda45-a30e-467e-bf44-126e88e59003","Type":"ContainerDied","Data":"5a72d8cb59114a90d91bd8cac392f692af9e355fba21ae3e2564df134e4639df"} Oct 30 07:48:42.976738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:42.976713 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:48:43.017945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.017909 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\" (UniqueName: \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\") pod \"d01bda45-a30e-467e-bf44-126e88e59003\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " Oct 30 07:48:43.018152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.018101 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\" (UniqueName: \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\") pod \"d01bda45-a30e-467e-bf44-126e88e59003\" (UID: \"d01bda45-a30e-467e-bf44-126e88e59003\") " Oct 30 07:48:43.018317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.018300 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.018386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.018327 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.018386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.018381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npt8v\" (UniqueName: \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.021157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.021124 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls" pod "d01bda45-a30e-467e-bf44-126e88e59003" (UID: "d01bda45-a30e-467e-bf44-126e88e59003"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:43.021873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.021844 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd" pod "d01bda45-a30e-467e-bf44-126e88e59003" (UID: "d01bda45-a30e-467e-bf44-126e88e59003"). InnerVolumeSpecName "kube-api-access-x26sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:43.133225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.133099 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.133868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.133821 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.135206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.135170 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npt8v\" (UniqueName: \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.135786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.135762 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26sd\" (UniqueName: \"kubernetes.io/projected/d01bda45-a30e-467e-bf44-126e88e59003-kube-api-access-x26sd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:43.135987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.135968 5894 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d01bda45-a30e-467e-bf44-126e88e59003-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:43.137595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.137568 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.153306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.153219 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.186617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.186490 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npt8v\" (UniqueName: \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\") pod \"maas-api-revoke-sub-dd1c48a1-m627v\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.250730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.241122 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:48:43.265700 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:43.265405 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb219d909_93ba_45fc_b293_bfd6fb34bd00.slice/crio-20ec5dcc2a9e3141e2f24a2e51e90f795594d10791640830996883f49b9c7edc WatchSource:0}: Error finding container 20ec5dcc2a9e3141e2f24a2e51e90f795594d10791640830996883f49b9c7edc: Status 404 returned error can't find the container with id 20ec5dcc2a9e3141e2f24a2e51e90f795594d10791640830996883f49b9c7edc Oct 30 07:48:43.295143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.294904 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:43.627526 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:43.627494 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96c5296_77ef_4424_aa09_e83dcf885329.slice/crio-e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d WatchSource:0}: Error finding container e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d: Status 404 returned error can't find the container with id e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d Oct 30 07:48:43.628944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.628922 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v"] Oct 30 07:48:43.687693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.687655 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:43.701093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.701068 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:43.848059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.848021 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:48:43.848821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.848788 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:43.849098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.849081 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d01bda45-a30e-467e-bf44-126e88e59003" containerName="sim" Oct 30 07:48:43.849163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.849100 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01bda45-a30e-467e-bf44-126e88e59003" containerName="sim" Oct 30 07:48:43.849260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.849251 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01bda45-a30e-467e-bf44-126e88e59003" containerName="sim" Oct 30 07:48:43.854852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.854827 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:43.871536 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.871505 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:48:43.968664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.968079 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhr6r\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:43.968664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.968185 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:43.968664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.968232 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:43.968664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.968358 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:43.981920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.981881 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" event={"ID":"d96c5296-77ef-4424-aa09-e83dcf885329","Type":"ContainerStarted","Data":"e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d"} Oct 30 07:48:43.984217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.984180 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" Oct 30 07:48:43.984466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.984186 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9" event={"ID":"d01bda45-a30e-467e-bf44-126e88e59003","Type":"ContainerDied","Data":"a92346721a25b414ccf85269623002c372972b2d5b8a3c4e2494c1be3ce5e16d"} Oct 30 07:48:43.984594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.984495 5894 scope.go:122] "RemoveContainer" containerID="5a72d8cb59114a90d91bd8cac392f692af9e355fba21ae3e2564df134e4639df" Oct 30 07:48:43.990435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.990384 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" event={"ID":"b219d909-93ba-45fc-b293-bfd6fb34bd00","Type":"ContainerStarted","Data":"926a365341f55515e370c3199eef4c5fc47d286905371e44ddf027f734e29f96"} Oct 30 07:48:43.990606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.990452 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" event={"ID":"b219d909-93ba-45fc-b293-bfd6fb34bd00","Type":"ContainerStarted","Data":"20ec5dcc2a9e3141e2f24a2e51e90f795594d10791640830996883f49b9c7edc"} Oct 30 07:48:43.990606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:43.990541 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:44.019385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.019358 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:48:44.069515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.069467 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhr6r\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.069762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.069586 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.069762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.069628 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.070112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.070078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.070576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.070550 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.073021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.070741 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.073456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.073435 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.091894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.091818 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" podStartSLOduration=2.091804134 podStartE2EDuration="2.091804134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:44.085581937 +0000 UTC m=+3087.776499375" watchObservedRunningTime="2026-10-30 07:48:44.091804134 +0000 UTC m=+3087.782721570" Oct 30 07:48:44.096789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.096753 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhr6r\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\") pod \"payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.129382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.129295 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:48:44.133599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.133568 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-e1f1c7/e2e-usage-sim-99fb7f-59fcf6f68d-2pfk9"] Oct 30 07:48:44.172388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.172359 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:44.324164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.323207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:44.561676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.538803 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01bda45-a30e-467e-bf44-126e88e59003" path="/var/lib/kubelet/pods/d01bda45-a30e-467e-bf44-126e88e59003/volumes" Oct 30 07:48:44.590382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.590337 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:48:44.605333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.590622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:44.647337 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:44.644623 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab45c68_5a3c_402a_aa23_f8ece1a4e635.slice/crio-b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7 WatchSource:0}: Error finding container b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7: Status 404 returned error can't find the container with id b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7 Oct 30 07:48:44.989661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:44.977698 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:48:45.038216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.038180 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd"] Oct 30 07:48:45.038532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.038314 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.167068 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cstzr\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.196924 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.197057 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.197343 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.190791 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd"] Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.197619 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:48:45.214038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.190885 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.242245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.241593 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" event={"ID":"0ab45c68-5a3c-402a-aa23-f8ece1a4e635","Type":"ContainerStarted","Data":"25b0e63a336b9af847e75ae6f141f9d515cff8ee7a3233496a1a156f584c86e5"} Oct 30 07:48:45.242245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.241695 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" event={"ID":"0ab45c68-5a3c-402a-aa23-f8ece1a4e635","Type":"ContainerStarted","Data":"b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7"} Oct 30 07:48:45.242245 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.241818 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:45.243872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.243834 5894 generic.go:356] "Generic (PLEG): container finished" podID="d96c5296-77ef-4424-aa09-e83dcf885329" containerID="56f239b7f6683b5be44ca919bab3e4061cd3e75b4ee79ab57a4d79089f5596be" exitCode=0 Oct 30 07:48:45.244224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.243871 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" event={"ID":"d96c5296-77ef-4424-aa09-e83dcf885329","Type":"ContainerDied","Data":"56f239b7f6683b5be44ca919bab3e4061cd3e75b4ee79ab57a4d79089f5596be"} Oct 30 07:48:45.298901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.298862 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.299115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299024 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxcdt\" (UniqueName: \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.299182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299128 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cstzr\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.299182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299167 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.299304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299231 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.299361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299323 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.299361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299352 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.299604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.299576 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.300068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.300041 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.302682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.302653 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.322682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.322656 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cstzr\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\") pod \"payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.343134 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.343084 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" podStartSLOduration=2.343068443 podStartE2EDuration="2.343068443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:45.339928263 +0000 UTC m=+3089.030845700" watchObservedRunningTime="2026-10-30 07:48:45.343068443 +0000 UTC m=+3089.033985879" Oct 30 07:48:45.402147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.402114 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.402331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.402176 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.402331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.402323 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxcdt\" (UniqueName: \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.402961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.402936 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.404468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.404448 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.426833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.426798 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxcdt\" (UniqueName: \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\") pod \"maas-api-revoke-sub-21e2467f-st8sd\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:45.543772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.543738 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:45.553920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:45.553888 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:46.011191 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:46.011145 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ca3eea_75d9_455f_9bd8_abf1f9e15868.slice/crio-52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f WatchSource:0}: Error finding container 52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f: Status 404 returned error can't find the container with id 52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f Oct 30 07:48:46.019660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.019609 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd"] Oct 30 07:48:46.047570 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:46.047508 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c0bf86_62e4_448a_a0fb_2d417082b998.slice/crio-635b135ab1e1ae7412434e8b8e2738a4bb68db871e6c2b5f9e6d945375613c2e WatchSource:0}: Error finding container 635b135ab1e1ae7412434e8b8e2738a4bb68db871e6c2b5f9e6d945375613c2e: Status 404 returned error can't find the container with id 635b135ab1e1ae7412434e8b8e2738a4bb68db871e6c2b5f9e6d945375613c2e Oct 30 07:48:46.049128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.049095 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:48:46.126764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.126737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:46.258317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.258284 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" event={"ID":"c2c0bf86-62e4-448a-a0fb-2d417082b998","Type":"ContainerStarted","Data":"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f"} Oct 30 07:48:46.258517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.258335 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" event={"ID":"c2c0bf86-62e4-448a-a0fb-2d417082b998","Type":"ContainerStarted","Data":"635b135ab1e1ae7412434e8b8e2738a4bb68db871e6c2b5f9e6d945375613c2e"} Oct 30 07:48:46.258517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.258383 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:46.259962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.259927 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" event={"ID":"19ca3eea-75d9-455f-9bd8-abf1f9e15868","Type":"ContainerStarted","Data":"52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f"} Oct 30 07:48:46.304734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.304621 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" podStartSLOduration=2.304604495 podStartE2EDuration="2.304604495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:46.300507604 +0000 UTC m=+3089.991425041" watchObservedRunningTime="2026-10-30 07:48:46.304604495 +0000 UTC m=+3089.995521931" Oct 30 07:48:46.410448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.410410 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:46.442180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.442150 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:48:46.443334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.443305 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d96c5296-77ef-4424-aa09-e83dcf885329" containerName="revoke-keys" Oct 30 07:48:46.443334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.443330 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d96c5296-77ef-4424-aa09-e83dcf885329" containerName="revoke-keys" Oct 30 07:48:46.443598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.443580 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d96c5296-77ef-4424-aa09-e83dcf885329" containerName="revoke-keys" Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\") pod \"d96c5296-77ef-4424-aa09-e83dcf885329\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462400 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462439 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\") pod \"d96c5296-77ef-4424-aa09-e83dcf885329\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462511 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\" (UniqueName: \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\") pod \"d96c5296-77ef-4424-aa09-e83dcf885329\" (UID: \"d96c5296-77ef-4424-aa09-e83dcf885329\") " Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462538 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.462925 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca" pod "d96c5296-77ef-4424-aa09-e83dcf885329" (UID: "d96c5296-77ef-4424-aa09-e83dcf885329"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:46.465685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.463815 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:46.466494 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.466466 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v" pod "d96c5296-77ef-4424-aa09-e83dcf885329" (UID: "d96c5296-77ef-4424-aa09-e83dcf885329"). InnerVolumeSpecName "kube-api-access-npt8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:46.467066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.467043 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca" pod "d96c5296-77ef-4424-aa09-e83dcf885329" (UID: "d96c5296-77ef-4424-aa09-e83dcf885329"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:46.565466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.565372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84nfq\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.565645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.565512 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.565775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.565756 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.565939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.565924 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.566045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.566032 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npt8v\" (UniqueName: \"kubernetes.io/projected/d96c5296-77ef-4424-aa09-e83dcf885329-kube-api-access-npt8v\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:46.566103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.566053 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d96c5296-77ef-4424-aa09-e83dcf885329-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:46.668033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668000 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.668193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668135 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.668193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668172 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84nfq\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.668323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668231 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.668873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668850 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.668980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.668891 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.670548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.670529 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.692601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.692568 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84nfq\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\") pod \"payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:46.801549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:46.801514 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:47.031451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.031420 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5"] Oct 30 07:48:47.035850 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.035822 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c34e81e-xdhq5"] Oct 30 07:48:47.111928 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:47.111893 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f60bfda_f30d_4a77_b6bc_4d6239955f7d.slice/crio-0ccea341bbc3a889ef7d834396e62927469a38d62f913d6c96b4f5acc01f0767 WatchSource:0}: Error finding container 0ccea341bbc3a889ef7d834396e62927469a38d62f913d6c96b4f5acc01f0767: Status 404 returned error can't find the container with id 0ccea341bbc3a889ef7d834396e62927469a38d62f913d6c96b4f5acc01f0767 Oct 30 07:48:47.113401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.113366 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:48:47.271831 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.271777 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" event={"ID":"7f60bfda-f30d-4a77-b6bc-4d6239955f7d","Type":"ContainerStarted","Data":"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a"} Oct 30 07:48:47.272020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.271843 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" event={"ID":"7f60bfda-f30d-4a77-b6bc-4d6239955f7d","Type":"ContainerStarted","Data":"0ccea341bbc3a889ef7d834396e62927469a38d62f913d6c96b4f5acc01f0767"} Oct 30 07:48:47.272020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.271874 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:47.273353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.273328 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" Oct 30 07:48:47.273497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.273326 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v" event={"ID":"d96c5296-77ef-4424-aa09-e83dcf885329","Type":"ContainerDied","Data":"e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d"} Oct 30 07:48:47.273497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.273449 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e99448c79a6fe7bd0bacfe2e03772564ff6edd265f25bc8f69cda1a494c8f27d" Oct 30 07:48:47.275263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.275233 5894 generic.go:356] "Generic (PLEG): container finished" podID="19ca3eea-75d9-455f-9bd8-abf1f9e15868" containerID="29aecdf0cdeb20a5babedbeb68af81deb177f3170f5839a2ddd0b3d94c0a426d" exitCode=0 Oct 30 07:48:47.275376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.275301 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" event={"ID":"19ca3eea-75d9-455f-9bd8-abf1f9e15868","Type":"ContainerDied","Data":"29aecdf0cdeb20a5babedbeb68af81deb177f3170f5839a2ddd0b3d94c0a426d"} Oct 30 07:48:47.314275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.314212 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" podStartSLOduration=1.314194259 podStartE2EDuration="1.314194259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:47.313803958 +0000 UTC m=+3091.004721394" watchObservedRunningTime="2026-10-30 07:48:47.314194259 +0000 UTC m=+3091.005111699" Oct 30 07:48:47.829044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:47.829007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:48.435828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.435796 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381c4e87-8621-415b-a5cc-1c072b5b90f9" path="/var/lib/kubelet/pods/381c4e87-8621-415b-a5cc-1c072b5b90f9/volumes" Oct 30 07:48:48.456303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.456278 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:48.499452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.499416 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\") pod \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " Oct 30 07:48:48.499630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.499555 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\" (UniqueName: \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\") pod \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " Oct 30 07:48:48.499630 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.499589 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\") pod \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\" (UID: \"19ca3eea-75d9-455f-9bd8-abf1f9e15868\") " Oct 30 07:48:48.500546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.500512 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca" pod "19ca3eea-75d9-455f-9bd8-abf1f9e15868" (UID: "19ca3eea-75d9-455f-9bd8-abf1f9e15868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:48.502253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.502221 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca" pod "19ca3eea-75d9-455f-9bd8-abf1f9e15868" (UID: "19ca3eea-75d9-455f-9bd8-abf1f9e15868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:48.502852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.502830 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt" pod "19ca3eea-75d9-455f-9bd8-abf1f9e15868" (UID: "19ca3eea-75d9-455f-9bd8-abf1f9e15868"). InnerVolumeSpecName "kube-api-access-fxcdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:48.602505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.602425 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:48.602505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.602458 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxcdt\" (UniqueName: \"kubernetes.io/projected/19ca3eea-75d9-455f-9bd8-abf1f9e15868-kube-api-access-fxcdt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:48.602505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:48.602470 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ca3eea-75d9-455f-9bd8-abf1f9e15868-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:49.308693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:49.308655 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" Oct 30 07:48:49.308908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:49.308666 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd" event={"ID":"19ca3eea-75d9-455f-9bd8-abf1f9e15868","Type":"ContainerDied","Data":"52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f"} Oct 30 07:48:49.308908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:49.308833 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a8019945835b31061393edbf9d9591d1a39c8d5ac048e717efd3e5f2483a8f" Oct 30 07:48:49.544777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:49.544742 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:51.123311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.123276 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:48:51.124210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.124192 5894 cpu_manager.go:414] "containerMap: removing container" podUID="19ca3eea-75d9-455f-9bd8-abf1f9e15868" containerName="revoke-keys" Oct 30 07:48:51.124263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.124211 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ca3eea-75d9-455f-9bd8-abf1f9e15868" containerName="revoke-keys" Oct 30 07:48:51.124362 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.124353 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ca3eea-75d9-455f-9bd8-abf1f9e15868" containerName="revoke-keys" Oct 30 07:48:51.128090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.128073 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.162931 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.162899 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:48:51.221540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.221510 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-759ae6-openshift-default-dockercfg-kbznf\"" Oct 30 07:48:51.242073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242038 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242074 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242277 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242337 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242428 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242555 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242541 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257rm\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242609 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242578 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242703 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.242852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.242837 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.329402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.329299 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:51.344753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.344709 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.344962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.344764 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.344962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.344902 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.344962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.344955 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345007 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345102 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257rm\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345151 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345198 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345216 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345400 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345512 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.345877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345848 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.346013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345874 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.346013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.345890 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.348086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.348058 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.348424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.348405 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.413518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.408353 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.413775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.413710 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-257rm\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\") pod \"e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.450920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.450883 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:51.779730 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:51.779703 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded868f9d_f92d_4189_a052_22bde008e5f9.slice/crio-0998b916f5b7b43360777333f993ca924a5e94d8b56244aff57d9d792cdd5bc4 WatchSource:0}: Error finding container 0998b916f5b7b43360777333f993ca924a5e94d8b56244aff57d9d792cdd5bc4: Status 404 returned error can't find the container with id 0998b916f5b7b43360777333f993ca924a5e94d8b56244aff57d9d792cdd5bc4 Oct 30 07:48:51.781967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.781941 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:48:51.782905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.782873 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:51.782951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.782938 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:51.782984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:51.782966 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:48:52.339887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.339847 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" event={"ID":"ed868f9d-f92d-4189-a052-22bde008e5f9","Type":"ContainerStarted","Data":"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78"} Oct 30 07:48:52.340399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.339945 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" event={"ID":"ed868f9d-f92d-4189-a052-22bde008e5f9","Type":"ContainerStarted","Data":"0998b916f5b7b43360777333f993ca924a5e94d8b56244aff57d9d792cdd5bc4"} Oct 30 07:48:52.413767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.413699 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podStartSLOduration=1.4136789140000001 podStartE2EDuration="1.413678914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:52.398733029 +0000 UTC m=+3096.089650483" watchObservedRunningTime="2026-10-30 07:48:52.413678914 +0000 UTC m=+3096.104596353" Oct 30 07:48:52.451473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.451438 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:52.453223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.453196 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:52.453379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:52.453246 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:53.012283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.012240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:53.146807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.146771 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:53.451411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.451377 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:53.451856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.451463 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:53.703181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.703093 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j"] Oct 30 07:48:53.710741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.710712 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.718001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.717965 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j"] Oct 30 07:48:53.726440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.726404 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:48:53.738590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.738552 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:48:53.738590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.738566 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:48:53.738794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.738676 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:48:53.814606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.814576 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7rqb\" (UniqueName: \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.814822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.814755 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.814875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.814855 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.916740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.916706 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7rqb\" (UniqueName: \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.916934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.916834 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.916934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.916898 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.917599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.917571 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.919247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.919228 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:53.940288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:53.940255 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7rqb\" (UniqueName: \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\") pod \"maas-api-revoke-sub-4e6c9cf0-6z97j\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:54.030895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:54.030861 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:54.399687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:54.399105 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j"] Oct 30 07:48:54.402675 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:48:54.402601 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec58f03_272f_44eb_88a4_6fef1d1fc9d8.slice/crio-547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0 WatchSource:0}: Error finding container 547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0: Status 404 returned error can't find the container with id 547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0 Oct 30 07:48:54.451775 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:54.451749 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:54.452102 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:54.451796 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:54.937233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:54.937196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:55.250847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.250759 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:48:55.330896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.330280 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:48:55.407531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.407491 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" event={"ID":"dec58f03-272f-44eb-88a4-6fef1d1fc9d8","Type":"ContainerStarted","Data":"deff1a7ea6406575624473abb64da3e73f0063a9e742daaf2ebbd729872f9c79"} Oct 30 07:48:55.407762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.407560 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" event={"ID":"dec58f03-272f-44eb-88a4-6fef1d1fc9d8","Type":"ContainerStarted","Data":"547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0"} Oct 30 07:48:55.451684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.451646 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:55.451858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.451713 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:55.453980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.453936 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" podStartSLOduration=2.453922503 podStartE2EDuration="2.453922503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:55.451731922 +0000 UTC m=+3099.142649391" watchObservedRunningTime="2026-10-30 07:48:55.453922503 +0000 UTC m=+3099.144839940" Oct 30 07:48:55.861929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.861836 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8"] Oct 30 07:48:55.867095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.867071 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:55.920990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.920959 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8"] Oct 30 07:48:55.959599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.959556 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:55.959798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.959629 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckp6f\" (UniqueName: \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:55.959798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:55.959758 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.061294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.061253 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.061479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.061378 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckp6f\" (UniqueName: \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.061625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.061606 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.062067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.062043 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.064123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.064098 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.088433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.088367 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckp6f\" (UniqueName: \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\") pod \"maas-api-revoke-sub-af70d619-h4pn8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.218113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.218025 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:56.264755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.264717 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:48:56.348256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.348211 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:48:56.425881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.425846 5894 generic.go:356] "Generic (PLEG): container finished" podID="dec58f03-272f-44eb-88a4-6fef1d1fc9d8" containerID="deff1a7ea6406575624473abb64da3e73f0063a9e742daaf2ebbd729872f9c79" exitCode=0 Oct 30 07:48:56.437845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.437809 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" event={"ID":"dec58f03-272f-44eb-88a4-6fef1d1fc9d8","Type":"ContainerDied","Data":"deff1a7ea6406575624473abb64da3e73f0063a9e742daaf2ebbd729872f9c79"} Oct 30 07:48:56.455289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.453057 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:56.462105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.462064 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:56.604151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.604113 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8"] Oct 30 07:48:56.760150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:56.760114 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:57.290663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.289829 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:48:57.414720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.413721 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:48:57.441719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.441672 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" event={"ID":"a55b0b65-9860-4d89-af4d-ebc585f254e8","Type":"ContainerStarted","Data":"a420df179d4aa58d5263c56411b94d0aa0130d41693c258ce2b60f01bcd51718"} Oct 30 07:48:57.441719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.441719 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" event={"ID":"a55b0b65-9860-4d89-af4d-ebc585f254e8","Type":"ContainerStarted","Data":"5be74e5339a44da545415d6aa3f38eccf6628c52438f2565a80ed5803bb6c6f1"} Oct 30 07:48:57.454685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.454586 5894 patch_prober.go:29] interesting pod/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Oct 30 07:48:57.454685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.454671 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Oct 30 07:48:57.507529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.507459 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" podStartSLOduration=2.507435747 podStartE2EDuration="2.507435747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:48:57.503911212 +0000 UTC m=+3101.194828654" watchObservedRunningTime="2026-10-30 07:48:57.507435747 +0000 UTC m=+3101.198353184" Oct 30 07:48:57.717360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.717332 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:57.843827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.843796 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\") pod \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " Oct 30 07:48:57.844011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.843995 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\" (UniqueName: \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\") pod \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " Oct 30 07:48:57.844113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.844098 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\") pod \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\" (UID: \"dec58f03-272f-44eb-88a4-6fef1d1fc9d8\") " Oct 30 07:48:57.844180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.844130 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca" pod "dec58f03-272f-44eb-88a4-6fef1d1fc9d8" (UID: "dec58f03-272f-44eb-88a4-6fef1d1fc9d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:57.844745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.844717 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:57.846124 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.846102 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb" pod "dec58f03-272f-44eb-88a4-6fef1d1fc9d8" (UID: "dec58f03-272f-44eb-88a4-6fef1d1fc9d8"). InnerVolumeSpecName "kube-api-access-f7rqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:57.846191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.846135 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca" pod "dec58f03-272f-44eb-88a4-6fef1d1fc9d8" (UID: "dec58f03-272f-44eb-88a4-6fef1d1fc9d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:57.946462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.946372 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7rqb\" (UniqueName: \"kubernetes.io/projected/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-kube-api-access-f7rqb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:57.946462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:57.946402 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec58f03-272f-44eb-88a4-6fef1d1fc9d8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:58.290533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.290505 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:48:58.378258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.376766 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:48:58.452352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.452322 5894 generic.go:356] "Generic (PLEG): container finished" podID="a55b0b65-9860-4d89-af4d-ebc585f254e8" containerID="a420df179d4aa58d5263c56411b94d0aa0130d41693c258ce2b60f01bcd51718" exitCode=0 Oct 30 07:48:58.452544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.452411 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" event={"ID":"a55b0b65-9860-4d89-af4d-ebc585f254e8","Type":"ContainerDied","Data":"a420df179d4aa58d5263c56411b94d0aa0130d41693c258ce2b60f01bcd51718"} Oct 30 07:48:58.454248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.454228 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" Oct 30 07:48:58.454378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.454267 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j" event={"ID":"dec58f03-272f-44eb-88a4-6fef1d1fc9d8","Type":"ContainerDied","Data":"547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0"} Oct 30 07:48:58.454378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.454312 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547ffa62392ef3ccb38f6676963390cbe1ceded070f7e3d89ae7f80e7c7cd7e0" Oct 30 07:48:58.457761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.457731 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:58.458018 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.457993 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:58.458832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.458811 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:48:58.637056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:58.636961 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:59.394929 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.394894 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:48:59.396315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.396251 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dec58f03-272f-44eb-88a4-6fef1d1fc9d8" containerName="revoke-keys" Oct 30 07:48:59.396315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.396285 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec58f03-272f-44eb-88a4-6fef1d1fc9d8" containerName="revoke-keys" Oct 30 07:48:59.396589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.396570 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec58f03-272f-44eb-88a4-6fef1d1fc9d8" containerName="revoke-keys" Oct 30 07:48:59.401804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.401777 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.421893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.421863 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:48:59.426734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.426706 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61f6c2\"/\"default-dockercfg-gmg2r\"" Oct 30 07:48:59.426974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.426836 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-61f6c2\"/\"test-model-61f6c2-kserve-self-signed-certs\"" Oct 30 07:48:59.426974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.426950 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61f6c2\"/\"openshift-service-ca.crt\"" Oct 30 07:48:59.429707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.429686 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-61f6c2\"/\"kube-root-ca.crt\"" Oct 30 07:48:59.445431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.445397 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:48:59.475648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.475593 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.476166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.475895 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.476166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.476009 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.476166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.476044 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.476166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.476093 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.476717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.476692 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n255b\" (UniqueName: \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.578749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.578710 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.578947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.578838 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.578947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.578903 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.578947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.578928 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.579164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.578966 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.579164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.579061 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n255b\" (UniqueName: \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.579987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.579951 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.580379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.580352 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.580880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.580823 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.582558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.582534 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.582934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.582889 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.730744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.709427 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n255b\" (UniqueName: \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\") pod \"test-model-61f6c2-kserve-c66b859ff-pvkkf\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.741672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.740890 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:48:59.810587 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.810544 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:48:59.891399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.891359 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\") pod \"a55b0b65-9860-4d89-af4d-ebc585f254e8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " Oct 30 07:48:59.891561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.891483 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\" (UniqueName: \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\") pod \"a55b0b65-9860-4d89-af4d-ebc585f254e8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " Oct 30 07:48:59.891561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.891515 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\") pod \"a55b0b65-9860-4d89-af4d-ebc585f254e8\" (UID: \"a55b0b65-9860-4d89-af4d-ebc585f254e8\") " Oct 30 07:48:59.892562 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.892529 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca" pod "a55b0b65-9860-4d89-af4d-ebc585f254e8" (UID: "a55b0b65-9860-4d89-af4d-ebc585f254e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:48:59.894441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.894415 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca" pod "a55b0b65-9860-4d89-af4d-ebc585f254e8" (UID: "a55b0b65-9860-4d89-af4d-ebc585f254e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:48:59.894539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.894472 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f" pod "a55b0b65-9860-4d89-af4d-ebc585f254e8" (UID: "a55b0b65-9860-4d89-af4d-ebc585f254e8"). InnerVolumeSpecName "kube-api-access-ckp6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:48:59.994499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.994415 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:59.994499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.994444 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckp6f\" (UniqueName: \"kubernetes.io/projected/a55b0b65-9860-4d89-af4d-ebc585f254e8-kube-api-access-ckp6f\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:48:59.994499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:48:59.994455 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55b0b65-9860-4d89-af4d-ebc585f254e8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:00.119427 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:00.119388 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cc35c21_a325_46d3_8fba_7fe5516a2fb1.slice/crio-dfccac8bf00eacbe75a4f1ecf98be1f391d0fc465ef005073776f4b91c8eb4b9 WatchSource:0}: Error finding container dfccac8bf00eacbe75a4f1ecf98be1f391d0fc465ef005073776f4b91c8eb4b9: Status 404 returned error can't find the container with id dfccac8bf00eacbe75a4f1ecf98be1f391d0fc465ef005073776f4b91c8eb4b9 Oct 30 07:49:00.121773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.121745 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:49:00.331476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.331423 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerName="payload-processing" containerID="cri-o://4a64138a514e4cd18add049e6f006ef183ea1690ed4f9a451eb26576ca00a3dd" gracePeriod=25 Oct 30 07:49:00.375064 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:00.375034 5894 logging.go:55] [core] [Channel #6211 SubChannel #6212]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.55:50052", ServerName: "10.134.1.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.55:50052: connect: connection refused" Oct 30 07:49:00.482115 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.482076 5894 generic.go:356] "Generic (PLEG): container finished" podID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerID="4a64138a514e4cd18add049e6f006ef183ea1690ed4f9a451eb26576ca00a3dd" exitCode=0 Oct 30 07:49:00.482564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.482534 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" event={"ID":"d4cb558c-6d46-40cb-b810-7c56f87dc3e1","Type":"ContainerDied","Data":"4a64138a514e4cd18add049e6f006ef183ea1690ed4f9a451eb26576ca00a3dd"} Oct 30 07:49:00.485573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.485525 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" event={"ID":"3cc35c21-a325-46d3-8fba-7fe5516a2fb1","Type":"ContainerStarted","Data":"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d"} Oct 30 07:49:00.485705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.485574 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" event={"ID":"3cc35c21-a325-46d3-8fba-7fe5516a2fb1","Type":"ContainerStarted","Data":"dfccac8bf00eacbe75a4f1ecf98be1f391d0fc465ef005073776f4b91c8eb4b9"} Oct 30 07:49:00.489899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.489873 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" Oct 30 07:49:00.490196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.490166 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8" event={"ID":"a55b0b65-9860-4d89-af4d-ebc585f254e8","Type":"ContainerDied","Data":"5be74e5339a44da545415d6aa3f38eccf6628c52438f2565a80ed5803bb6c6f1"} Oct 30 07:49:00.490285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.490225 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be74e5339a44da545415d6aa3f38eccf6628c52438f2565a80ed5803bb6c6f1" Oct 30 07:49:00.548748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.548702 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:00.550627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.550571 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" podStartSLOduration=1.550555809 podStartE2EDuration="1.550555809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:00.541145953 +0000 UTC m=+3104.232063416" watchObservedRunningTime="2026-10-30 07:49:00.550555809 +0000 UTC m=+3104.241473249" Oct 30 07:49:00.663745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.663697 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:49:00.704393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.704348 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\") pod \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " Oct 30 07:49:00.704575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.704414 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\") pod \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " Oct 30 07:49:00.704720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.704574 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\") pod \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " Oct 30 07:49:00.704720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.704666 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\") pod \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\" (UID: \"d4cb558c-6d46-40cb-b810-7c56f87dc3e1\") " Oct 30 07:49:00.705160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.705124 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume" pod "d4cb558c-6d46-40cb-b810-7c56f87dc3e1" (UID: "d4cb558c-6d46-40cb-b810-7c56f87dc3e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:00.705339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.705302 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume" pod "d4cb558c-6d46-40cb-b810-7c56f87dc3e1" (UID: "d4cb558c-6d46-40cb-b810-7c56f87dc3e1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:00.705519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.705499 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:00.705822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.705548 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:00.713806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.708142 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials" pod "d4cb558c-6d46-40cb-b810-7c56f87dc3e1" (UID: "d4cb558c-6d46-40cb-b810-7c56f87dc3e1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:00.713806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.709070 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64" pod "d4cb558c-6d46-40cb-b810-7c56f87dc3e1" (UID: "d4cb558c-6d46-40cb-b810-7c56f87dc3e1"). InnerVolumeSpecName "kube-api-access-lmk64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:00.743184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.743148 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:49:00.806691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.806627 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmk64\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-kube-api-access-lmk64\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:00.806691 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:00.806686 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cb558c-6d46-40cb-b810-7c56f87dc3e1-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:01.349897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.349854 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerName="payload-processing" containerID="cri-o://03149e4864cf13d645cb9d168d1952f30ff79fcafc1c5c000c3d3ad62b2230ea" gracePeriod=25 Oct 30 07:49:01.374553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.374510 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.55:50052\" within 1s: context deadline exceeded" Oct 30 07:49:01.390121 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:01.390101 5894 logging.go:55] [core] [Channel #6213 SubChannel #6214]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.56:50052", ServerName: "10.134.1.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.56:50052: connect: connection refused" Oct 30 07:49:01.502026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.501988 5894 generic.go:356] "Generic (PLEG): container finished" podID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerID="03149e4864cf13d645cb9d168d1952f30ff79fcafc1c5c000c3d3ad62b2230ea" exitCode=0 Oct 30 07:49:01.502220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.502066 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" event={"ID":"1f3c06c6-79cc-459b-990a-3547ee71b5b7","Type":"ContainerDied","Data":"03149e4864cf13d645cb9d168d1952f30ff79fcafc1c5c000c3d3ad62b2230ea"} Oct 30 07:49:01.503991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.503969 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" Oct 30 07:49:01.503991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.503976 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295" event={"ID":"d4cb558c-6d46-40cb-b810-7c56f87dc3e1","Type":"ContainerDied","Data":"3a923966f4c0b3bb9bf6b93bd5b3e9907973cbb3fa0cc74229eabf3c078dacd2"} Oct 30 07:49:01.504169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.504039 5894 scope.go:122] "RemoveContainer" containerID="4a64138a514e4cd18add049e6f006ef183ea1690ed4f9a451eb26576ca00a3dd" Oct 30 07:49:01.561520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.561489 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:49:01.571921 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.571882 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-ffbfcn295"] Oct 30 07:49:01.663776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.663748 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:49:01.719985 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.719947 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\") pod \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " Oct 30 07:49:01.720180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.720009 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\") pod \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " Oct 30 07:49:01.720180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.720077 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\") pod \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " Oct 30 07:49:01.720306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.720290 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\") pod \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\" (UID: \"1f3c06c6-79cc-459b-990a-3547ee71b5b7\") " Oct 30 07:49:01.720480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.720445 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume" pod "1f3c06c6-79cc-459b-990a-3547ee71b5b7" (UID: "1f3c06c6-79cc-459b-990a-3547ee71b5b7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:01.721057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.721031 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume" pod "1f3c06c6-79cc-459b-990a-3547ee71b5b7" (UID: "1f3c06c6-79cc-459b-990a-3547ee71b5b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:01.721265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.721239 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:01.722804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.722777 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l" pod "1f3c06c6-79cc-459b-990a-3547ee71b5b7" (UID: "1f3c06c6-79cc-459b-990a-3547ee71b5b7"). InnerVolumeSpecName "kube-api-access-68k7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:01.723261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.723229 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials" pod "1f3c06c6-79cc-459b-990a-3547ee71b5b7" (UID: "1f3c06c6-79cc-459b-990a-3547ee71b5b7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:01.822970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.822931 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3c06c6-79cc-459b-990a-3547ee71b5b7-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:01.822970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.822971 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68k7l\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-kube-api-access-68k7l\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:01.823430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:01.822985 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1f3c06c6-79cc-459b-990a-3547ee71b5b7-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:02.387139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.386039 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:02.389920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.389868 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.56:50052\" within 1s: context deadline exceeded" Oct 30 07:49:02.414777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.414736 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" podUID="c8446c48-b973-4fc9-afa5-a2d42a157412" containerName="payload-processing" containerID="cri-o://76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002" gracePeriod=25 Oct 30 07:49:02.444052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.444017 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" path="/var/lib/kubelet/pods/d4cb558c-6d46-40cb-b810-7c56f87dc3e1/volumes" Oct 30 07:49:02.517176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.517147 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" Oct 30 07:49:02.517371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.517191 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc" event={"ID":"1f3c06c6-79cc-459b-990a-3547ee71b5b7","Type":"ContainerDied","Data":"25d4e4b41e62aceac067a4f9fba8b761ddf69182c444c71ad521c283821704d8"} Oct 30 07:49:02.517371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.517265 5894 scope.go:122] "RemoveContainer" containerID="03149e4864cf13d645cb9d168d1952f30ff79fcafc1c5c000c3d3ad62b2230ea" Oct 30 07:49:02.575418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.575381 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:49:02.585280 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.585247 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-748fqfzzc"] Oct 30 07:49:02.729712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.729643 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:49:02.810185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.810142 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:02.856807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.856765 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\") pod \"c8446c48-b973-4fc9-afa5-a2d42a157412\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " Oct 30 07:49:02.857332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857033 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\") pod \"c8446c48-b973-4fc9-afa5-a2d42a157412\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " Oct 30 07:49:02.857332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857112 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\") pod \"c8446c48-b973-4fc9-afa5-a2d42a157412\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " Oct 30 07:49:02.857332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857129 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume" pod "c8446c48-b973-4fc9-afa5-a2d42a157412" (UID: "c8446c48-b973-4fc9-afa5-a2d42a157412"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:02.857332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857225 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\") pod \"c8446c48-b973-4fc9-afa5-a2d42a157412\" (UID: \"c8446c48-b973-4fc9-afa5-a2d42a157412\") " Oct 30 07:49:02.857565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857479 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume" pod "c8446c48-b973-4fc9-afa5-a2d42a157412" (UID: "c8446c48-b973-4fc9-afa5-a2d42a157412"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:02.857915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857896 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:02.857975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.857916 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c8446c48-b973-4fc9-afa5-a2d42a157412-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:02.859422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.859395 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd" pod "c8446c48-b973-4fc9-afa5-a2d42a157412" (UID: "c8446c48-b973-4fc9-afa5-a2d42a157412"). InnerVolumeSpecName "kube-api-access-47qjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:02.859557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.859434 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials" pod "c8446c48-b973-4fc9-afa5-a2d42a157412" (UID: "c8446c48-b973-4fc9-afa5-a2d42a157412"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:02.970344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.967181 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:02.970344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:02.967219 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qjd\" (UniqueName: \"kubernetes.io/projected/c8446c48-b973-4fc9-afa5-a2d42a157412-kube-api-access-47qjd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:03.387704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.387380 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" podUID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" containerName="payload-processing" containerID="cri-o://ec148d7fcef96b4f5fbab260125551a9e98a96653897994ad905edc55b26aad3" gracePeriod=25 Oct 30 07:49:03.531951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.531903 5894 generic.go:356] "Generic (PLEG): container finished" podID="c8446c48-b973-4fc9-afa5-a2d42a157412" containerID="76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002" exitCode=0 Oct 30 07:49:03.532150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.531959 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" Oct 30 07:49:03.532150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.532002 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" event={"ID":"c8446c48-b973-4fc9-afa5-a2d42a157412","Type":"ContainerDied","Data":"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002"} Oct 30 07:49:03.532150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.532078 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml" event={"ID":"c8446c48-b973-4fc9-afa5-a2d42a157412","Type":"ContainerDied","Data":"92f4edde77f21289027523b51a2ee890c10878ec48f6ef55fc4b3da148135b82"} Oct 30 07:49:03.532150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.532111 5894 scope.go:122] "RemoveContainer" containerID="76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002" Oct 30 07:49:03.536285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.536257 5894 generic.go:356] "Generic (PLEG): container finished" podID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" containerID="ec148d7fcef96b4f5fbab260125551a9e98a96653897994ad905edc55b26aad3" exitCode=0 Oct 30 07:49:03.536395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.536340 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" event={"ID":"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7","Type":"ContainerDied","Data":"ec148d7fcef96b4f5fbab260125551a9e98a96653897994ad905edc55b26aad3"} Oct 30 07:49:03.546723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.546690 5894 scope.go:122] "RemoveContainer" containerID="76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002" Oct 30 07:49:03.547127 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:49:03.547097 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002\": container with ID starting with 76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002 not found: ID does not exist" containerID="76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002" Oct 30 07:49:03.547218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.547140 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002"} err="failed to get container status \"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002\": rpc error: code = NotFound desc = could not find container \"76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002\": container with ID starting with 76c2c6dcef2a03b03fc812f57580e01f36b1b2219a0f749267e04cc1d7d4b002 not found: ID does not exist" Oct 30 07:49:03.606711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.606616 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:49:03.613612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.613577 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-5df6x4fml"] Oct 30 07:49:03.679692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.679665 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:49:03.791567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.791526 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\") pod \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " Oct 30 07:49:03.791823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.791662 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\") pod \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " Oct 30 07:49:03.791823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.791699 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\") pod \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " Oct 30 07:49:03.791941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.791872 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\") pod \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\" (UID: \"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7\") " Oct 30 07:49:03.794173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.794125 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume" pod "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" (UID: "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:03.794173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.794139 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume" pod "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" (UID: "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:03.797232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.797163 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785" pod "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" (UID: "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7"). InnerVolumeSpecName "kube-api-access-9h785". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:03.797232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.797208 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials" pod "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" (UID: "6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:03.894218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.894180 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h785\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-kube-api-access-9h785\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:03.894898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.894873 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:03.895040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.895004 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:03.895171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:03.895157 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:04.352322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.352280 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:04.452329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.452294 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" path="/var/lib/kubelet/pods/1f3c06c6-79cc-459b-990a-3547ee71b5b7/volumes" Oct 30 07:49:04.453021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.452994 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8446c48-b973-4fc9-afa5-a2d42a157412" path="/var/lib/kubelet/pods/c8446c48-b973-4fc9-afa5-a2d42a157412/volumes" Oct 30 07:49:04.546327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.546295 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" Oct 30 07:49:04.546327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.546310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4" event={"ID":"6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7","Type":"ContainerDied","Data":"e48a1dc0609c4ccb808027ff7183e53d0876ad71f15d4fc3666fc11a4b15d582"} Oct 30 07:49:04.546607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.546370 5894 scope.go:122] "RemoveContainer" containerID="ec148d7fcef96b4f5fbab260125551a9e98a96653897994ad905edc55b26aad3" Oct 30 07:49:04.608194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.608106 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:49:04.620046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:04.620018 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-94bf4rks4"] Oct 30 07:49:04.830519 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:49:04.830476 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:49:05.409532 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:05.409498 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:49:05.498336 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:05.498298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:05.635721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:05.635669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:06.138250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.138211 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:06.413048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.412945 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:06.435896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.435859 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" path="/var/lib/kubelet/pods/6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7/volumes" Oct 30 07:49:06.483381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.483338 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:49:06.484942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.484911 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerName="payload-processing" Oct 30 07:49:06.484942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.484940 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.484976 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a55b0b65-9860-4d89-af4d-ebc585f254e8" containerName="revoke-keys" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.484985 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55b0b65-9860-4d89-af4d-ebc585f254e8" containerName="revoke-keys" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485008 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485016 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485055 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485063 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485092 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c8446c48-b973-4fc9-afa5-a2d42a157412" containerName="payload-processing" Oct 30 07:49:06.485168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485099 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8446c48-b973-4fc9-afa5-a2d42a157412" containerName="payload-processing" Oct 30 07:49:06.485599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485321 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3c06c6-79cc-459b-990a-3547ee71b5b7" containerName="payload-processing" Oct 30 07:49:06.485599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485343 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe6447b-aba6-44f4-8ee3-a1dcfb460dd7" containerName="payload-processing" Oct 30 07:49:06.485599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485358 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8446c48-b973-4fc9-afa5-a2d42a157412" containerName="payload-processing" Oct 30 07:49:06.485599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485378 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cb558c-6d46-40cb-b810-7c56f87dc3e1" containerName="payload-processing" Oct 30 07:49:06.485599 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.485390 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55b0b65-9860-4d89-af4d-ebc585f254e8" containerName="revoke-keys" Oct 30 07:49:06.492798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.492773 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.508127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.508098 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:49:06.512991 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.512957 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc3a23\"/\"default-dockercfg-274qr\"" Oct 30 07:49:06.513152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.513071 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc3a23\"/\"openshift-service-ca.crt\"" Oct 30 07:49:06.521373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.521329 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-fc3a23\"/\"kube-root-ca.crt\"" Oct 30 07:49:06.523517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.523490 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-fc3a23\"/\"test-model-fc3a23-kserve-self-signed-certs\"" Oct 30 07:49:06.533381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.533353 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:06.650442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650390 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.650442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650447 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.650757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650518 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.650757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650696 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szcd\" (UniqueName: \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.650757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650737 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.650948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.650767 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.755833 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szcd\" (UniqueName: \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.755905 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.755936 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.756163 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.756216 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.756272 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.757710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.757226 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.758246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.757830 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.758246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.757902 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.787154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.784277 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.795657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.793246 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.814902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.813373 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szcd\" (UniqueName: \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\") pod \"test-model-fc3a23-kserve-8655746c79-2h4z5\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:06.814902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:06.813706 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:07.206884 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:07.206853 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod860fba1b_d47b_46bb_85a5_48b1536660a6.slice/crio-116b8c1c33cdd9f548d40b35ae582df89ef4f8cf0c0a93991b770708039fa046 WatchSource:0}: Error finding container 116b8c1c33cdd9f548d40b35ae582df89ef4f8cf0c0a93991b770708039fa046: Status 404 returned error can't find the container with id 116b8c1c33cdd9f548d40b35ae582df89ef4f8cf0c0a93991b770708039fa046 Oct 30 07:49:07.208848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.208816 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:49:07.392091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.392053 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:49:07.473962 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.473872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:07.582656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.582594 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" event={"ID":"860fba1b-d47b-46bb-85a5-48b1536660a6","Type":"ContainerStarted","Data":"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d"} Oct 30 07:49:07.582886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.582698 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" event={"ID":"860fba1b-d47b-46bb-85a5-48b1536660a6","Type":"ContainerStarted","Data":"116b8c1c33cdd9f548d40b35ae582df89ef4f8cf0c0a93991b770708039fa046"} Oct 30 07:49:07.637210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.637136 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" podStartSLOduration=1.637115633 podStartE2EDuration="1.637115633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:07.63025134 +0000 UTC m=+3111.321168778" watchObservedRunningTime="2026-10-30 07:49:07.637115633 +0000 UTC m=+3111.328033071" Oct 30 07:49:07.827658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:07.824939 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:08.249233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:08.249148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:08.408758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:08.405226 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:08.709370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:08.709337 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:49:08.757058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:08.757026 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:08.951334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:08.951303 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:09.764342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:09.746941 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:49:09.901538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:09.896446 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:49:10.033200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.033160 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp"] Oct 30 07:49:10.037620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.037588 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8b664ca-wsvjp"] Oct 30 07:49:10.105934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.105896 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:49:10.184244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.184212 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:10.409926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.409835 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" podUID="b219d909-93ba-45fc-b293-bfd6fb34bd00" containerName="payload-processing" containerID="cri-o://926a365341f55515e370c3199eef4c5fc47d286905371e44ddf027f734e29f96" gracePeriod=25 Oct 30 07:49:10.437311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.437270 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caddd582-2419-4fe6-a557-2d5ce033a0d6" path="/var/lib/kubelet/pods/caddd582-2419-4fe6-a557-2d5ce033a0d6/volumes" Oct 30 07:49:10.455845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.455807 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:10.672064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.671081 5894 generic.go:356] "Generic (PLEG): container finished" podID="b219d909-93ba-45fc-b293-bfd6fb34bd00" containerID="926a365341f55515e370c3199eef4c5fc47d286905371e44ddf027f734e29f96" exitCode=0 Oct 30 07:49:10.673730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.673416 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" event={"ID":"b219d909-93ba-45fc-b293-bfd6fb34bd00","Type":"ContainerDied","Data":"926a365341f55515e370c3199eef4c5fc47d286905371e44ddf027f734e29f96"} Oct 30 07:49:10.695655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.694179 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:49:10.911659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:10.911194 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:49:11.101246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.100936 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\") pod \"b219d909-93ba-45fc-b293-bfd6fb34bd00\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " Oct 30 07:49:11.101716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.101670 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\") pod \"b219d909-93ba-45fc-b293-bfd6fb34bd00\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " Oct 30 07:49:11.102037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.102001 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\") pod \"b219d909-93ba-45fc-b293-bfd6fb34bd00\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " Oct 30 07:49:11.102486 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.102442 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\") pod \"b219d909-93ba-45fc-b293-bfd6fb34bd00\" (UID: \"b219d909-93ba-45fc-b293-bfd6fb34bd00\") " Oct 30 07:49:11.106922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.106659 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc" pod "b219d909-93ba-45fc-b293-bfd6fb34bd00" (UID: "b219d909-93ba-45fc-b293-bfd6fb34bd00"). InnerVolumeSpecName "kube-api-access-5xclc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:11.107515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.107253 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume" pod "b219d909-93ba-45fc-b293-bfd6fb34bd00" (UID: "b219d909-93ba-45fc-b293-bfd6fb34bd00"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:11.107928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.107889 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume" pod "b219d909-93ba-45fc-b293-bfd6fb34bd00" (UID: "b219d909-93ba-45fc-b293-bfd6fb34bd00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:11.111312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.111279 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials" pod "b219d909-93ba-45fc-b293-bfd6fb34bd00" (UID: "b219d909-93ba-45fc-b293-bfd6fb34bd00"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:11.205683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.205283 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xclc\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-kube-api-access-5xclc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:11.205683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.205331 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:11.205683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.205349 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b219d909-93ba-45fc-b293-bfd6fb34bd00-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:11.205683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.205365 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b219d909-93ba-45fc-b293-bfd6fb34bd00-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:11.683826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.683793 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" Oct 30 07:49:11.684047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.683834 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg" event={"ID":"b219d909-93ba-45fc-b293-bfd6fb34bd00","Type":"ContainerDied","Data":"20ec5dcc2a9e3141e2f24a2e51e90f795594d10791640830996883f49b9c7edc"} Oct 30 07:49:11.684047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.683899 5894 scope.go:122] "RemoveContainer" containerID="926a365341f55515e370c3199eef4c5fc47d286905371e44ddf027f734e29f96" Oct 30 07:49:11.742729 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.742691 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:49:11.754883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:11.754850 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-705337/payload-processing-external-model-e2e-extmod-a-705337-6759bpqpg"] Oct 30 07:49:12.144601 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.144568 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:12.392795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.392750 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" podUID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" containerName="payload-processing" containerID="cri-o://25b0e63a336b9af847e75ae6f141f9d515cff8ee7a3233496a1a156f584c86e5" gracePeriod=25 Oct 30 07:49:12.396754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.396687 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:12.434484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.434451 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b219d909-93ba-45fc-b293-bfd6fb34bd00" path="/var/lib/kubelet/pods/b219d909-93ba-45fc-b293-bfd6fb34bd00/volumes" Oct 30 07:49:12.517741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.517699 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:12.712015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.711977 5894 generic.go:356] "Generic (PLEG): container finished" podID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" containerID="25b0e63a336b9af847e75ae6f141f9d515cff8ee7a3233496a1a156f584c86e5" exitCode=0 Oct 30 07:49:12.712214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.712054 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" event={"ID":"0ab45c68-5a3c-402a-aa23-f8ece1a4e635","Type":"ContainerDied","Data":"25b0e63a336b9af847e75ae6f141f9d515cff8ee7a3233496a1a156f584c86e5"} Oct 30 07:49:12.712214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.712128 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" event={"ID":"0ab45c68-5a3c-402a-aa23-f8ece1a4e635","Type":"ContainerDied","Data":"b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7"} Oct 30 07:49:12.712214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.712145 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9c53358d15c0874bbb2ccbb90646617a691dad6651ba4b48682e45a49d141e7" Oct 30 07:49:12.714369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.714345 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:49:12.741750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.741721 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\") pod \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " Oct 30 07:49:12.741902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.741795 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\") pod \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " Oct 30 07:49:12.741902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.741815 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\") pod \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " Oct 30 07:49:12.742037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.742020 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\") pod \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\" (UID: \"0ab45c68-5a3c-402a-aa23-f8ece1a4e635\") " Oct 30 07:49:12.742332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.742305 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume" pod "0ab45c68-5a3c-402a-aa23-f8ece1a4e635" (UID: "0ab45c68-5a3c-402a-aa23-f8ece1a4e635"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:12.742438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.742362 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume" pod "0ab45c68-5a3c-402a-aa23-f8ece1a4e635" (UID: "0ab45c68-5a3c-402a-aa23-f8ece1a4e635"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:12.742802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.742779 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:12.742802 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.742800 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:12.744164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.744133 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r" pod "0ab45c68-5a3c-402a-aa23-f8ece1a4e635" (UID: "0ab45c68-5a3c-402a-aa23-f8ece1a4e635"). InnerVolumeSpecName "kube-api-access-zhr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:12.744284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.744220 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials" pod "0ab45c68-5a3c-402a-aa23-f8ece1a4e635" (UID: "0ab45c68-5a3c-402a-aa23-f8ece1a4e635"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:12.844519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.844483 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhr6r\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-kube-api-access-zhr6r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:12.844519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.844516 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ab45c68-5a3c-402a-aa23-f8ece1a4e635-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:12.911024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:12.910985 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:13.419384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.419343 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:13.661715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.653135 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:13.715249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.710720 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" podUID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" containerName="payload-processing" containerID="cri-o://87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a" gracePeriod=25 Oct 30 07:49:13.756418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.756383 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg" Oct 30 07:49:13.858611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.858575 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:49:13.876333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:13.876194 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bffe43/payload-processing-external-model-e2e-extmod-b-bffe43-5bdfrztjg"] Oct 30 07:49:14.057913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.057882 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:49:14.150222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.150188 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:14.190055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.183176 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\") pod \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " Oct 30 07:49:14.190055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.183285 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\") pod \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " Oct 30 07:49:14.190055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.183366 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\") pod \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " Oct 30 07:49:14.190055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.183615 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\") pod \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\" (UID: \"7f60bfda-f30d-4a77-b6bc-4d6239955f7d\") " Oct 30 07:49:14.207683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.206941 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume" pod "7f60bfda-f30d-4a77-b6bc-4d6239955f7d" (UID: "7f60bfda-f30d-4a77-b6bc-4d6239955f7d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:14.218333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.218288 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume" pod "7f60bfda-f30d-4a77-b6bc-4d6239955f7d" (UID: "7f60bfda-f30d-4a77-b6bc-4d6239955f7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:14.225233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.225170 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials" pod "7f60bfda-f30d-4a77-b6bc-4d6239955f7d" (UID: "7f60bfda-f30d-4a77-b6bc-4d6239955f7d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:14.238321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.238280 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq" pod "7f60bfda-f30d-4a77-b6bc-4d6239955f7d" (UID: "7f60bfda-f30d-4a77-b6bc-4d6239955f7d"). InnerVolumeSpecName "kube-api-access-84nfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:14.286448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.286408 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:14.286448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.286444 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:14.286448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.286455 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84nfq\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-kube-api-access-84nfq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:14.286774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.286464 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f60bfda-f30d-4a77-b6bc-4d6239955f7d-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:14.438151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.438110 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" path="/var/lib/kubelet/pods/0ab45c68-5a3c-402a-aa23-f8ece1a4e635/volumes" Oct 30 07:49:14.767137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.767041 5894 generic.go:356] "Generic (PLEG): container finished" podID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" containerID="87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a" exitCode=0 Oct 30 07:49:14.767137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.767094 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" Oct 30 07:49:14.767137 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.767124 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" event={"ID":"7f60bfda-f30d-4a77-b6bc-4d6239955f7d","Type":"ContainerDied","Data":"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a"} Oct 30 07:49:14.767443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.767183 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn" event={"ID":"7f60bfda-f30d-4a77-b6bc-4d6239955f7d","Type":"ContainerDied","Data":"0ccea341bbc3a889ef7d834396e62927469a38d62f913d6c96b4f5acc01f0767"} Oct 30 07:49:14.767443 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.767208 5894 scope.go:122] "RemoveContainer" containerID="87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a" Oct 30 07:49:14.781367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.781343 5894 scope.go:122] "RemoveContainer" containerID="87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a" Oct 30 07:49:14.781719 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:49:14.781686 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a\": container with ID starting with 87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a not found: ID does not exist" containerID="87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a" Oct 30 07:49:14.781790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.781730 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a"} err="failed to get container status \"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a\": rpc error: code = NotFound desc = could not find container \"87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a\": container with ID starting with 87f663d3b482088b00ac0cf5c1e588c6ff0ccecd9a83b33702896bf073bf772a not found: ID does not exist" Oct 30 07:49:14.828660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.828603 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:49:14.840306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:14.840270 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-b3a7d7/payload-processing-external-model-e2e-extmod-b-b3a7d7-5b7d57hgn"] Oct 30 07:49:15.106589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.106542 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" podUID="c2c0bf86-62e4-448a-a0fb-2d417082b998" containerName="payload-processing" containerID="cri-o://e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f" gracePeriod=25 Oct 30 07:49:15.253390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.253352 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:15.372145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.372012 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:15.447007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.446970 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.507203 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\") pod \"c2c0bf86-62e4-448a-a0fb-2d417082b998\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.507306 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\") pod \"c2c0bf86-62e4-448a-a0fb-2d417082b998\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.507351 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\") pod \"c2c0bf86-62e4-448a-a0fb-2d417082b998\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.507582 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\") pod \"c2c0bf86-62e4-448a-a0fb-2d417082b998\" (UID: \"c2c0bf86-62e4-448a-a0fb-2d417082b998\") " Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.507735 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume" pod "c2c0bf86-62e4-448a-a0fb-2d417082b998" (UID: "c2c0bf86-62e4-448a-a0fb-2d417082b998"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.508190 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume" pod "c2c0bf86-62e4-448a-a0fb-2d417082b998" (UID: "c2c0bf86-62e4-448a-a0fb-2d417082b998"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.508949 5894 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-routing-overlay-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.508976 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2c0bf86-62e4-448a-a0fb-2d417082b998-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:15.512658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.511148 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials" pod "c2c0bf86-62e4-448a-a0fb-2d417082b998" (UID: "c2c0bf86-62e4-448a-a0fb-2d417082b998"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:15.532786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.520395 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr" pod "c2c0bf86-62e4-448a-a0fb-2d417082b998" (UID: "c2c0bf86-62e4-448a-a0fb-2d417082b998"). InnerVolumeSpecName "kube-api-access-cstzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:49:15.611059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.611021 5894 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-provider-credentials\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:15.611059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.611055 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cstzr\" (UniqueName: \"kubernetes.io/projected/c2c0bf86-62e4-448a-a0fb-2d417082b998-kube-api-access-cstzr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:49:15.738107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.737997 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:15.777716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.777672 5894 generic.go:356] "Generic (PLEG): container finished" podID="c2c0bf86-62e4-448a-a0fb-2d417082b998" containerID="e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f" exitCode=0 Oct 30 07:49:15.777935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.777743 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" Oct 30 07:49:15.777935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.777745 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" event={"ID":"c2c0bf86-62e4-448a-a0fb-2d417082b998","Type":"ContainerDied","Data":"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f"} Oct 30 07:49:15.777935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.777822 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n" event={"ID":"c2c0bf86-62e4-448a-a0fb-2d417082b998","Type":"ContainerDied","Data":"635b135ab1e1ae7412434e8b8e2738a4bb68db871e6c2b5f9e6d945375613c2e"} Oct 30 07:49:15.777935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.777844 5894 scope.go:122] "RemoveContainer" containerID="e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f" Oct 30 07:49:15.791515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.791494 5894 scope.go:122] "RemoveContainer" containerID="e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f" Oct 30 07:49:15.791849 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:49:15.791824 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f\": container with ID starting with e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f not found: ID does not exist" containerID="e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f" Oct 30 07:49:15.791945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.791859 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f"} err="failed to get container status \"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f\": rpc error: code = NotFound desc = could not find container \"e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f\": container with ID starting with e976cb6bbfbff906208e511f4f1f0aae93efa1664d9c0d5ab3de83dd8e18689f not found: ID does not exist" Oct 30 07:49:15.833159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.833126 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:49:15.851141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:15.846873 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-a91a83/payload-processing-external-model-e2e-extmod-a-a91a83-66c9rt48n"] Oct 30 07:49:16.035730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.034063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:16.237916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.237870 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:16.520756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.508090 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" path="/var/lib/kubelet/pods/7f60bfda-f30d-4a77-b6bc-4d6239955f7d/volumes" Oct 30 07:49:16.520756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.508823 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c0bf86-62e4-448a-a0fb-2d417082b998" path="/var/lib/kubelet/pods/c2c0bf86-62e4-448a-a0fb-2d417082b998/volumes" Oct 30 07:49:16.690036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.689999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:16.815105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.815069 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:16.828448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:16.828420 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:17.812739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:17.812709 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:49:17.884158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:17.884130 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:18.025778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:18.025744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:19.588696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.588656 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:19.703738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.703704 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:19.767196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.767160 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:49:19.768000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.767983 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" containerName="payload-processing" Oct 30 07:49:19.768068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768001 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" containerName="payload-processing" Oct 30 07:49:19.768068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768028 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c2c0bf86-62e4-448a-a0fb-2d417082b998" containerName="payload-processing" Oct 30 07:49:19.768068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768033 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c0bf86-62e4-448a-a0fb-2d417082b998" containerName="payload-processing" Oct 30 07:49:19.768068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768060 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b219d909-93ba-45fc-b293-bfd6fb34bd00" containerName="payload-processing" Oct 30 07:49:19.768068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768066 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b219d909-93ba-45fc-b293-bfd6fb34bd00" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768077 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768082 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768201 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b219d909-93ba-45fc-b293-bfd6fb34bd00" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768211 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f60bfda-f30d-4a77-b6bc-4d6239955f7d" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768222 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab45c68-5a3c-402a-aa23-f8ece1a4e635" containerName="payload-processing" Oct 30 07:49:19.768255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.768232 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c0bf86-62e4-448a-a0fb-2d417082b998" containerName="payload-processing" Oct 30 07:49:19.785997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.785957 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:19.808133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.808095 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:49:19.808795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.808766 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-057cfa\"" Oct 30 07:49:19.858678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.858560 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:19.858960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.858933 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:19.971719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.868135 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:49:20.035061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:19.883624 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-057cfa-dockercfg-krnfn\"" Oct 30 07:49:20.035313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.018278 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:20.035471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.035452 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:20.041151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.037813 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:20.120217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.110118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:20.120217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.118136 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\") pod \"payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:20.120217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.119110 5894 scope.go:122] "RemoveContainer" containerID="41ab1dc87e086ef86d88c833ba6b2773cd161b584e5fc017be66a54ebdbf8126" Oct 30 07:49:20.127476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.127426 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.128174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.128152 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:49:20.128277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.128213 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc"] Oct 30 07:49:20.128277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.128235 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85cbd019-vntqc"] Oct 30 07:49:20.150756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.150727 5894 scope.go:122] "RemoveContainer" containerID="2246ab4f183a14ef000fc00189537e08f981bf45238d67843aa38433d9861332" Oct 30 07:49:20.166912 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.166870 5894 scope.go:122] "RemoveContainer" containerID="43dd825126ce0ad9f7a5fed9f39f71d4b92dc31432df122d6e358e43a195b228" Oct 30 07:49:20.200039 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.195945 5894 scope.go:122] "RemoveContainer" containerID="e51dc912644d5089793c83603fda53319eaba192eac4c568c12294d869e2298a" Oct 30 07:49:20.239603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.239576 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv5wk\" (UniqueName: \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.239821 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.239804 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.249408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.249373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:20.347721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.347681 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.349763 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.349735 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv5wk\" (UniqueName: \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.350367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.349408 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.414716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.394619 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv5wk\" (UniqueName: \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\") pod \"payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.415064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.414854 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:20.460657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.458313 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:20.468940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.468403 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af4b0c1f-4e27-47bf-83ea-2bc91bdb9077" path="/var/lib/kubelet/pods/af4b0c1f-4e27-47bf-83ea-2bc91bdb9077/volumes" Oct 30 07:49:20.790046 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:20.790005 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ef68d9_11ac_4808_849a_fb681a4b01a2.slice/crio-1c8a405c0d1e6b51395788f4bcfec0affa3741b59f5c71d672d0c5dc9382d48a WatchSource:0}: Error finding container 1c8a405c0d1e6b51395788f4bcfec0affa3741b59f5c71d672d0c5dc9382d48a: Status 404 returned error can't find the container with id 1c8a405c0d1e6b51395788f4bcfec0affa3741b59f5c71d672d0c5dc9382d48a Oct 30 07:49:20.791156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.791131 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:49:20.812942 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:20.812909 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37550a32_f812_464b_8543_4631f9461985.slice/crio-0a5d5b31790ca9502cc91b5fb8b06060d3208d0d5a873bb4deab54154179d0f2 WatchSource:0}: Error finding container 0a5d5b31790ca9502cc91b5fb8b06060d3208d0d5a873bb4deab54154179d0f2: Status 404 returned error can't find the container with id 0a5d5b31790ca9502cc91b5fb8b06060d3208d0d5a873bb4deab54154179d0f2 Oct 30 07:49:20.814248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.814219 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:49:20.861300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.861252 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" event={"ID":"37550a32-f812-464b-8543-4631f9461985","Type":"ContainerStarted","Data":"0a5d5b31790ca9502cc91b5fb8b06060d3208d0d5a873bb4deab54154179d0f2"} Oct 30 07:49:20.863347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:20.863316 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" event={"ID":"40ef68d9-11ac-4808-849a-fb681a4b01a2","Type":"ContainerStarted","Data":"1c8a405c0d1e6b51395788f4bcfec0affa3741b59f5c71d672d0c5dc9382d48a"} Oct 30 07:49:21.873023 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.872980 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" event={"ID":"40ef68d9-11ac-4808-849a-fb681a4b01a2","Type":"ContainerStarted","Data":"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90"} Oct 30 07:49:21.873521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.873056 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:21.875331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.875303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" event={"ID":"37550a32-f812-464b-8543-4631f9461985","Type":"ContainerStarted","Data":"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f"} Oct 30 07:49:21.875471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.875432 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:21.886726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.886692 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:21.919185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.919116 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" podStartSLOduration=2.919096242 podStartE2EDuration="2.919096242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:21.912959968 +0000 UTC m=+3125.603877404" watchObservedRunningTime="2026-10-30 07:49:21.919096242 +0000 UTC m=+3125.610013679" Oct 30 07:49:21.955074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:21.955004 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" podStartSLOduration=2.954984548 podStartE2EDuration="2.954984548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:21.954274357 +0000 UTC m=+3125.645191794" watchObservedRunningTime="2026-10-30 07:49:21.954984548 +0000 UTC m=+3125.645902034" Oct 30 07:49:22.006834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.006797 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:49:22.013003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.012974 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.015156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.015122 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:22.028524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.028490 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51147f\"" Oct 30 07:49:22.029500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.029470 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:49:22.080307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.080266 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:49:22.085814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.085786 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.087299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.087271 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.087436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.087398 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmvtg\" (UniqueName: \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.103554 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.103521 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51147f-dockercfg-qhrdm\"" Oct 30 07:49:22.117899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.117864 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:49:22.196421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.196324 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmvtg\" (UniqueName: \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.212548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.205789 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.212548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.205950 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.212548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.206216 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.212548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.207329 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.272969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.251869 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:22.333656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.322386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.334014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.333979 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.347963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.342446 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.381661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.358944 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmvtg\" (UniqueName: \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\") pod \"payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.427900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.427863 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\") pod \"payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.634370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.634327 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:22.706746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.706718 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:22.945178 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:22.945144 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode923e6a7_aced_439e_81d2_df142d5f7c82.slice/crio-dc624cb888b4ab7719b593bb893da873f57f255f74dcde259df21801f65e8d69 WatchSource:0}: Error finding container dc624cb888b4ab7719b593bb893da873f57f255f74dcde259df21801f65e8d69: Status 404 returned error can't find the container with id dc624cb888b4ab7719b593bb893da873f57f255f74dcde259df21801f65e8d69 Oct 30 07:49:22.946988 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:22.946963 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:49:23.029818 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:23.029780 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5636aab6_f0d3_4afd_9029_7529e2c130b8.slice/crio-193102b35eb20046a0009869625b4dcbd630571141c557a348edae6ef85507c5 WatchSource:0}: Error finding container 193102b35eb20046a0009869625b4dcbd630571141c557a348edae6ef85507c5: Status 404 returned error can't find the container with id 193102b35eb20046a0009869625b4dcbd630571141c557a348edae6ef85507c5 Oct 30 07:49:23.031225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.031202 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:49:23.866996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.866957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:23.937616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.936980 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" event={"ID":"e923e6a7-aced-439e-81d2-df142d5f7c82","Type":"ContainerStarted","Data":"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf"} Oct 30 07:49:23.937616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.937050 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" event={"ID":"e923e6a7-aced-439e-81d2-df142d5f7c82","Type":"ContainerStarted","Data":"dc624cb888b4ab7719b593bb893da873f57f255f74dcde259df21801f65e8d69"} Oct 30 07:49:23.937616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.937106 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:23.950206 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.950169 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" event={"ID":"5636aab6-f0d3-4afd-9029-7529e2c130b8","Type":"ContainerStarted","Data":"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad"} Oct 30 07:49:23.950773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.950235 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" event={"ID":"5636aab6-f0d3-4afd-9029-7529e2c130b8","Type":"ContainerStarted","Data":"193102b35eb20046a0009869625b4dcbd630571141c557a348edae6ef85507c5"} Oct 30 07:49:23.950773 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.950391 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:23.980329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:23.980275 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" podStartSLOduration=2.980261916 podStartE2EDuration="2.980261916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:23.980051532 +0000 UTC m=+3127.670968971" watchObservedRunningTime="2026-10-30 07:49:23.980261916 +0000 UTC m=+3127.671179353" Oct 30 07:49:24.024466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:24.024412 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" podStartSLOduration=2.024397962 podStartE2EDuration="2.024397962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:24.02027253 +0000 UTC m=+3127.711189967" watchObservedRunningTime="2026-10-30 07:49:24.024397962 +0000 UTC m=+3127.715315433" Oct 30 07:49:25.619524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:25.619486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:27.268782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:27.268741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:28.938456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:28.938409 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:29.860130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.860094 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:49:29.866887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.866859 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.885925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.885878 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:49:29.891360 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.891333 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-759ae6\"/\"default-dockercfg-hzxbh\"" Oct 30 07:49:29.891540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.891407 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-759ae6\"/\"openshift-service-ca.crt\"" Oct 30 07:49:29.894418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.894397 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-759ae6\"/\"test-model-759ae6-kserve-self-signed-certs\"" Oct 30 07:49:29.894564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.894546 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-759ae6\"/\"kube-root-ca.crt\"" Oct 30 07:49:29.922573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.922538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:29.965020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.964980 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.965447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.965220 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.965447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.965364 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.965447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.965400 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.965651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.965498 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbl7k\" (UniqueName: \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:29.965651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:29.965572 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.067670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.067607 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.067880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.067760 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.067880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.067794 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.067880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.067831 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbl7k\" (UniqueName: \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.067880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.067866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.068133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.068067 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.068133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.068102 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.068405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.068371 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.068662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.068627 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.070888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.070860 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.071087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.071063 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.105657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.096328 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbl7k\" (UniqueName: \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\") pod \"test-model-759ae6-kserve-558db6dccc-wpfkr\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.195731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.194941 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:30.602351 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:30.602317 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1ab6e2d_08c2_4b2e_8992_5ca0e6df46b9.slice/crio-4531b8ce1e48c9f5806a1a6ab4817b13f5c9d51a59c6a68ec559a477e564ccfd WatchSource:0}: Error finding container 4531b8ce1e48c9f5806a1a6ab4817b13f5c9d51a59c6a68ec559a477e564ccfd: Status 404 returned error can't find the container with id 4531b8ce1e48c9f5806a1a6ab4817b13f5c9d51a59c6a68ec559a477e564ccfd Oct 30 07:49:30.603857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.603834 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:49:30.642559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:30.642526 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:31.045034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:31.044991 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" event={"ID":"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9","Type":"ContainerStarted","Data":"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc"} Oct 30 07:49:31.045438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:31.045042 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" event={"ID":"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9","Type":"ContainerStarted","Data":"4531b8ce1e48c9f5806a1a6ab4817b13f5c9d51a59c6a68ec559a477e564ccfd"} Oct 30 07:49:31.090735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:31.090682 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" podStartSLOduration=2.090666587 podStartE2EDuration="2.090666587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:31.086904245 +0000 UTC m=+3134.777821682" watchObservedRunningTime="2026-10-30 07:49:31.090666587 +0000 UTC m=+3134.781584031" Oct 30 07:49:31.225658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:31.221509 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:31.579695 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:31.579660 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:32.338774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:32.338729 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:32.887356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:32.887328 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:49:32.887557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:32.887394 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:49:33.035212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:33.035179 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2"] Oct 30 07:49:33.039263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:33.039237 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6052151-q8ng2"] Oct 30 07:49:34.108198 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:34.108163 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:34.436768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:34.436679 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e60fc68-daa8-4499-8586-5ca3c27e3771" path="/var/lib/kubelet/pods/7e60fc68-daa8-4499-8586-5ca3c27e3771/volumes" Oct 30 07:49:34.831810 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:49:34.831762 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:49:34.962619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:34.962584 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:49:35.903086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:35.903043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:35.976778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:35.976741 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:49:37.678251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:37.678215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:38.333686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:38.333629 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:49:39.353784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:39.353744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:40.195864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:40.195825 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:40.213112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:40.213078 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:41.158329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:41.158294 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:49:41.432229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:41.432148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:43.192451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:43.192414 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:44.980506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:44.980467 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:46.818136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:46.818098 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:48.461207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:48.461167 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:50.165884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:50.165850 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:51.737237 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:51.737188 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:53.396345 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:53.396308 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:55.234160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:55.234119 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:56.471315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.471279 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:49:56.477910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.477888 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.495552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.495508 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d5f604-openshift-default-dockercfg-mkjm6\"" Oct 30 07:49:56.515174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.515131 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:49:56.538195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538118 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.538527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538202 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.538527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538267 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.538527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538344 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdgj\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.538527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538378 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.539209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538593 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.539209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538620 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.539209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538729 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.539209 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.538766 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640537 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdgj\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640589 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640757 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640788 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640852 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.640946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640889 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.640952 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641017 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641078 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641171 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641481 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641677 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641780 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.641983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.641960 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.643622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.643601 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.643884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.643865 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.664403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.664366 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.664569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.664459 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdgj\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\") pod \"e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.795136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.795093 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:56.856933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:56.856743 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:57.122140 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:49:57.122107 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2474bd6_f0df_4748_9059_5510490cbcce.slice/crio-fc2d08103c83fa40d1e3b7ea5fe3e95f20013629fb60c11b8bd3a528da1054ea WatchSource:0}: Error finding container fc2d08103c83fa40d1e3b7ea5fe3e95f20013629fb60c11b8bd3a528da1054ea: Status 404 returned error can't find the container with id fc2d08103c83fa40d1e3b7ea5fe3e95f20013629fb60c11b8bd3a528da1054ea Oct 30 07:49:57.124190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.124160 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:49:57.125356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.125321 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:49:57.125446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.125387 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:49:57.125446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.125418 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:49:57.351878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.351834 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" event={"ID":"a2474bd6-f0df-4748-9059-5510490cbcce","Type":"ContainerStarted","Data":"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318"} Oct 30 07:49:57.352065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.351906 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" event={"ID":"a2474bd6-f0df-4748-9059-5510490cbcce","Type":"ContainerStarted","Data":"fc2d08103c83fa40d1e3b7ea5fe3e95f20013629fb60c11b8bd3a528da1054ea"} Oct 30 07:49:57.400401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.400288 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podStartSLOduration=1.400272027 podStartE2EDuration="1.400272027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:49:57.396963656 +0000 UTC m=+3161.087881093" watchObservedRunningTime="2026-10-30 07:49:57.400272027 +0000 UTC m=+3161.091189464" Oct 30 07:49:57.796502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:57.796156 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:49:58.720147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:58.720009 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:49:58.808686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:58.798795 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body= Oct 30 07:49:58.808686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:58.798877 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" Oct 30 07:49:59.796119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:59.796071 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body= Oct 30 07:49:59.796361 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:49:59.796129 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" Oct 30 07:50:00.539618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:00.539583 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:00.796269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:00.796171 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body= Oct 30 07:50:00.796269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:00.796244 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" Oct 30 07:50:01.796092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:01.796060 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body= Oct 30 07:50:01.796580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:01.796556 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" Oct 30 07:50:02.210307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:02.210213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:02.796091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:02.796052 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" start-of-body= Oct 30 07:50:02.796667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:02.796109 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.83:15021/healthz/ready\": dial tcp 10.134.1.83:15021: connect: connection refused" Oct 30 07:50:03.802588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:03.802553 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:50:03.803118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:03.803087 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:50:03.804015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:03.803992 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:50:04.011240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:04.011199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:04.833413 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:50:04.833368 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:50:05.702208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:05.702170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:07.324410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:07.324369 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:08.947895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:08.947857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:10.622322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:10.622285 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:12.207440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:12.207404 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:13.871111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:13.871066 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:13.988589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:13.988549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:16.713595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:16.713565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:17.422304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.422264 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:50:17.427558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.427536 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.449223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.449186 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:50:17.454762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.454725 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 07:50:17.454762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.454725 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 07:50:17.455036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.454903 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 07:50:17.513225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.513175 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:17.546333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.546297 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-g865r\"" Oct 30 07:50:17.546531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.546401 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-51147f-serving-cert\"" Oct 30 07:50:17.554351 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.554313 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.554602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.554571 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfk9b\" (UniqueName: \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.554815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.554794 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.554897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.554869 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.632104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.632060 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:17.676920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.673576 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfk9b\" (UniqueName: \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.676920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.673787 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.676920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.673856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.676920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.674032 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.692748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.677962 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.700656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.696382 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.721924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.721887 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.736098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.736015 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfk9b\" (UniqueName: \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\") pod \"maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.746130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.746091 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:17.755120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:17.755086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:18.065392 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:50:18.065364 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ab756e_5453_465e_bbd3_055ffb3f10a4.slice/crio-05c942b08000e37082a938aff7236d5cb18376a6693b85dd9ea22bd8145bcfa4 WatchSource:0}: Error finding container 05c942b08000e37082a938aff7236d5cb18376a6693b85dd9ea22bd8145bcfa4: Status 404 returned error can't find the container with id 05c942b08000e37082a938aff7236d5cb18376a6693b85dd9ea22bd8145bcfa4 Oct 30 07:50:18.066984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:18.066962 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 07:50:18.067202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:18.067181 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:50:18.658661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:18.658601 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" event={"ID":"d3ab756e-5453-465e-bbd3-055ffb3f10a4","Type":"ContainerStarted","Data":"05c942b08000e37082a938aff7236d5cb18376a6693b85dd9ea22bd8145bcfa4"} Oct 30 07:50:19.345388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:19.345290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:20.491480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.491440 5894 scope.go:122] "RemoveContainer" containerID="ab3d272233c8d73e981b270e1fbcb12639ab68255af2e08308286a7e2ccda3c2" Oct 30 07:50:20.508906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.508874 5894 scope.go:122] "RemoveContainer" containerID="37b2e50fbe9238ed92bf0e44b9da410c024ccbce8d76c2aea923f26acf17eb8d" Oct 30 07:50:20.519547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.519525 5894 scope.go:122] "RemoveContainer" containerID="dff15ee0a57c52437187d19d81654f982e8a8c304d3522879c232d0bad9d2a0b" Oct 30 07:50:20.530223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.530195 5894 scope.go:122] "RemoveContainer" containerID="0efd45ea88337281d92fa5f1edfdefbb1d210b64175f38e816f9470293c3faf0" Oct 30 07:50:20.681154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.681113 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" event={"ID":"d3ab756e-5453-465e-bbd3-055ffb3f10a4","Type":"ContainerStarted","Data":"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181"} Oct 30 07:50:20.681344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.681221 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:20.724824 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:20.724755 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" podStartSLOduration=2.064163222 podStartE2EDuration="3.724735861s" totalImagesPullingTime="1.660572639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:50:20.724079634 +0000 UTC m=+3184.414997071" watchObservedRunningTime="2026-10-30 07:50:20.724735861 +0000 UTC m=+3184.415653299" Oct 30 07:50:21.074615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:21.074573 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:22.810863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:22.810829 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:24.741796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:24.741761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:26.427714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:26.427677 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:26.696557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:26.696474 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:50:28.058820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:28.058784 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:29.738399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:29.738367 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:31.361648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:31.361609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:31.702735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:31.702593 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:33.026844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:33.026799 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:34.726227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:34.726196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:34.876763 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:50:34.876725 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:50:36.387204 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:36.387168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:38.042582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:38.042552 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:39.711974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:39.711930 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:41.321758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:41.321718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:43.031698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:43.031626 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:44.723782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:44.723741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:46.303015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:46.302979 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:47.820660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:47.804057 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:50:48.080379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:48.080292 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:49.749772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:49.749736 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:51.360284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:51.360242 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:53.140886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:53.140836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:54.883616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:54.883574 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:56.579471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:56.579402 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:50:58.339087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:50:58.339030 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:00.055481 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:00.055431 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:01.658037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:01.658001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:03.439808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:03.439769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:04.831099 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:51:04.831058 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:51:05.133158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:05.133075 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:06.839842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:06.839806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:08.542449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:08.542411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:10.177748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:10.177707 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:11.909908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:11.909871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:13.690054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:13.690014 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:15.507923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:15.507887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:15.701126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:15.701087 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:16.190828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:16.190787 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:16.690614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:16.690564 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:17.558293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:17.558254 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:17.685749 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:17.685713 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:17.907356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:17.907265 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:18.406908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.406872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:18.851948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.851897 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 07:51:18.888158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.888125 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:18.937655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.925456 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:18.938352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.932586 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-268576-openshift-default-dockercfg-2kmx4\"" Oct 30 07:51:18.999014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.972708 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.001272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.001239 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.001492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.001471 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.001664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.001646 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.001941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.001922 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.002094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.002079 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.002236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.002222 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.002445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.002414 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9kbt\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.032678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.032615 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.033133 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:18.999624 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 07:51:19.155263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.155171 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.158110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.158081 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.158771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.158748 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.158982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.158968 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.159157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.159143 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.165364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.165333 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.165619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.165597 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.165909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.165890 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9kbt\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.166082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.166069 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.166903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.166881 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.172662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.159789 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.173783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.173760 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.174197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.160045 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.174340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.160249 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.176786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.176763 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.182397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.182369 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.237500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.237465 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.238949 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.238922 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9kbt\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\") pod \"e2e-inf-b-268576-openshift-default-d598b979c-d2z8k\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.276671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.276617 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:19.572297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.572214 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:19.627357 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:51:19.627316 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd898a2f_7a0f_47b3_b134_cdc3bd15590a.slice/crio-653f8b0baefa4c788c54b2e6fc492971d921636713d357db07b930a0bf9bfa68 WatchSource:0}: Error finding container 653f8b0baefa4c788c54b2e6fc492971d921636713d357db07b930a0bf9bfa68: Status 404 returned error can't find the container with id 653f8b0baefa4c788c54b2e6fc492971d921636713d357db07b930a0bf9bfa68 Oct 30 07:51:19.628181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.628155 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 07:51:19.630686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.630624 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:51:19.630772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.630733 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:51:19.630772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.630764 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:51:19.793473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:19.793437 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:20.446903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.446863 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerStarted","Data":"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5"} Oct 30 07:51:20.447366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.446920 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerStarted","Data":"653f8b0baefa4c788c54b2e6fc492971d921636713d357db07b930a0bf9bfa68"} Oct 30 07:51:20.510433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.510359 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podStartSLOduration=2.510337898 podStartE2EDuration="2.510337898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:20.505137729 +0000 UTC m=+3244.196055167" watchObservedRunningTime="2026-10-30 07:51:20.510337898 +0000 UTC m=+3244.201255336" Oct 30 07:51:20.577874 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.577840 5894 scope.go:122] "RemoveContainer" containerID="acf91373d2d52a9e48b7509f5361b2aebe5fdc819479fb2d45e11e5069fbe2b9" Oct 30 07:51:20.686960 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.686919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:20.821625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:20.821583 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:21.277705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:21.277671 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:21.286067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:21.286035 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:21.455490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:21.455456 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:21.456452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:21.456433 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:51:21.825122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:21.825086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:22.468088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:22.468032 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:22.598166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:22.598119 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:22.972714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:22.972363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:23.085473 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:23.085435 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:23.829009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:23.828975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:24.736111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.736050 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:24.861798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.861759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:24.909864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.909830 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 07:51:24.916616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.916594 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:24.935801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.935774 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 07:51:24.940920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.940892 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-268576\"" Oct 30 07:51:24.949397 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:24.949370 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-268576-dockercfg-hkfdp\"" Oct 30 07:51:25.133045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.133003 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.133371 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.133346 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpm2b\" (UniqueName: \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.249255 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.249212 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.249603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.249582 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpm2b\" (UniqueName: \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.252029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.252004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.281687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.281596 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpm2b\" (UniqueName: \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\") pod \"payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.295012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.294974 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 07:51:25.301882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.301798 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 07:51:25.301882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.301914 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.453080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.452990 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.453252 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.453098 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2dr\" (UniqueName: \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.546172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.546139 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:25.555143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.555109 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.555304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.555227 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2dr\" (UniqueName: \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.555857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.555835 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.579035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.579004 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2dr\" (UniqueName: \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\") pod \"payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.631799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.631000 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:25.672216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.672185 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:25.869122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.869065 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 07:51:25.870762 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:51:25.870630 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fe8038_db5d_4ce7_85df_584226b4e777.slice/crio-2932e435547b94929092609b1fcfa426063bc7e0d114abacad4f9b774e097248 WatchSource:0}: Error finding container 2932e435547b94929092609b1fcfa426063bc7e0d114abacad4f9b774e097248: Status 404 returned error can't find the container with id 2932e435547b94929092609b1fcfa426063bc7e0d114abacad4f9b774e097248 Oct 30 07:51:25.982127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:25.982104 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 07:51:25.982509 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:51:25.982479 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7423ac5b_14cf_4d34_a97a_f0156336a7c6.slice/crio-8314ee76411d638a296970efbca3096ce4971a827c5f8f74f2aa8f0fabb07157 WatchSource:0}: Error finding container 8314ee76411d638a296970efbca3096ce4971a827c5f8f74f2aa8f0fabb07157: Status 404 returned error can't find the container with id 8314ee76411d638a296970efbca3096ce4971a827c5f8f74f2aa8f0fabb07157 Oct 30 07:51:26.503247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.503209 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" event={"ID":"d7fe8038-db5d-4ce7-85df-584226b4e777","Type":"ContainerStarted","Data":"7c231430f8df26fc70948574e883499113e624bf0e6e9f9989748434effefdb1"} Oct 30 07:51:26.503463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.503261 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" event={"ID":"d7fe8038-db5d-4ce7-85df-584226b4e777","Type":"ContainerStarted","Data":"2932e435547b94929092609b1fcfa426063bc7e0d114abacad4f9b774e097248"} Oct 30 07:51:26.503463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.503292 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:26.504986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.504955 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" event={"ID":"7423ac5b-14cf-4d34-a97a-f0156336a7c6","Type":"ContainerStarted","Data":"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9"} Oct 30 07:51:26.505119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.504997 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" event={"ID":"7423ac5b-14cf-4d34-a97a-f0156336a7c6","Type":"ContainerStarted","Data":"8314ee76411d638a296970efbca3096ce4971a827c5f8f74f2aa8f0fabb07157"} Oct 30 07:51:26.505119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.505031 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:26.578195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.578138 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" podStartSLOduration=2.5781237299999997 podStartE2EDuration="2.57812373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:26.572063187 +0000 UTC m=+3250.262980626" watchObservedRunningTime="2026-10-30 07:51:26.57812373 +0000 UTC m=+3250.269041167" Oct 30 07:51:26.615518 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:26.615460 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" podStartSLOduration=2.615446433 podStartE2EDuration="2.615446433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:51:26.61289437 +0000 UTC m=+3250.303811807" watchObservedRunningTime="2026-10-30 07:51:26.615446433 +0000 UTC m=+3250.306363849" Oct 30 07:51:27.429930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:27.429896 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:29.088876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:29.088837 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:29.520576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:29.520481 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:30.838088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:30.838046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:31.819865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:31.819833 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:32.170013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:32.169931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:32.582524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:32.578096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:34.223388 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:34.223352 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:34.831863 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:51:34.831825 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:51:35.964036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:35.964001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:37.525794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:37.525765 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 07:51:37.526271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:37.525832 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 07:51:37.694174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:37.694138 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:39.311809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:39.311772 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:39.779956 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:39.779915 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:51:41.123654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:41.123596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:42.920179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:42.920139 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:44.637653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:44.637603 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:46.432692 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:46.432629 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:48.035075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:48.035036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:49.710954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:49.710911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:51.433028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:51.432984 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:53.098998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:53.098958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:54.712484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:54.712446 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:56.349007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:56.348962 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:56.743008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:56.742919 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:51:56.743195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:56.743163 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" containerID="cri-o://03fbddea6e07473661f7fc8150314cc0856a8a1fc3dec35031f3a100b29e778c" gracePeriod=30 Oct 30 07:51:58.038423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:58.038387 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:51:59.775224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:51:59.775189 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:01.148982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:01.148932 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:52:01.149426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:01.149009 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:52:01.528151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:01.528116 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:01.917496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:01.917463 5894 generic.go:356] "Generic (PLEG): container finished" podID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerID="03fbddea6e07473661f7fc8150314cc0856a8a1fc3dec35031f3a100b29e778c" exitCode=0 Oct 30 07:52:01.917728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:01.917535 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" event={"ID":"09f1bc3b-6c56-4146-b73e-0cf5dda18116","Type":"ContainerDied","Data":"03fbddea6e07473661f7fc8150314cc0856a8a1fc3dec35031f3a100b29e778c"} Oct 30 07:52:02.049610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.049584 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:52:02.124658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124541 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.124856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124661 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.124856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.124856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124804 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124893 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.124934 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:02.125045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125023 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:02.125045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125036 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125067 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125196 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125245 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\") pod \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\" (UID: \"09f1bc3b-6c56-4146-b73e-0cf5dda18116\") " Oct 30 07:52:02.125457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125288 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:02.125851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125602 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:02.125851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.125822 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:02.126202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.126173 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.126299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.126206 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.126299 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.126282 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.126429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.126299 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.126429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.126313 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.128257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.128230 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:02.129506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.129472 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "kube-api-access-rcnhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:02.130717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.130001 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:52:02.130717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.130203 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token" pod "09f1bc3b-6c56-4146-b73e-0cf5dda18116" (UID: "09f1bc3b-6c56-4146-b73e-0cf5dda18116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:02.232053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.232018 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnhm\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-kube-api-access-rcnhm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.232549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.232052 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.232549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.232092 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.232549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.232104 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09f1bc3b-6c56-4146-b73e-0cf5dda18116-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:02.928152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.928113 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" Oct 30 07:52:02.928152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.928120 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn" event={"ID":"09f1bc3b-6c56-4146-b73e-0cf5dda18116","Type":"ContainerDied","Data":"5dda0f9bea8cb7b74283cd5f735f4c55971ca32e85a2ef68283d329225a306aa"} Oct 30 07:52:02.928404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.928196 5894 scope.go:122] "RemoveContainer" containerID="03fbddea6e07473661f7fc8150314cc0856a8a1fc3dec35031f3a100b29e778c" Oct 30 07:52:02.995704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:02.995670 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:52:03.014399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:03.014366 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-k2fhn"] Oct 30 07:52:03.171550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:03.171511 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:04.435409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:04.435370 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" path="/var/lib/kubelet/pods/09f1bc3b-6c56-4146-b73e-0cf5dda18116/volumes" Oct 30 07:52:04.832860 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:52:04.832817 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:52:04.882009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:04.881973 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:05.135796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:05.135716 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:52:05.135963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:05.135925 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" containerID="cri-o://6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57" gracePeriod=30 Oct 30 07:52:06.693851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:06.693622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:07.162222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.162182 5894 patch_prober.go:29] interesting pod/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:52:07.162426 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.162253 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:52:07.735982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.735946 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84"] Oct 30 07:52:07.736866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.736849 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" Oct 30 07:52:07.736866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.736867 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" Oct 30 07:52:07.737049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.737040 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f1bc3b-6c56-4146-b73e-0cf5dda18116" containerName="istio-proxy" Oct 30 07:52:07.743006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.742984 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.759094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.759064 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:52:07.760250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.760221 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84"] Oct 30 07:52:07.767328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.767304 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:52:07.820890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.820854 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvjhn\" (UniqueName: \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.821072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.821009 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.821072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.821034 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.923259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.923217 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvjhn\" (UniqueName: \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.923449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.923327 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.923449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.923350 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.924061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.924032 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.925830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.925805 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:07.952757 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:07.952724 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvjhn\" (UniqueName: \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\") pod \"maas-api-revoke-sub-93702fbb-mbg84\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:08.061323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:08.061290 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:08.373040 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:52:08.373006 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd6bedb_1158_401b_b376_2e8eda058a7d.slice/crio-04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5 WatchSource:0}: Error finding container 04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5: Status 404 returned error can't find the container with id 04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5 Oct 30 07:52:08.373619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:08.373595 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84"] Oct 30 07:52:08.426576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:08.426551 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:09.007894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:09.007804 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" event={"ID":"7cd6bedb-1158-401b-b376-2e8eda058a7d","Type":"ContainerStarted","Data":"e2477cceb712d69e92563f7990312bcfc94e8fadc1fe7b81c3ac59b2381561f0"} Oct 30 07:52:09.007894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:09.007862 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" event={"ID":"7cd6bedb-1158-401b-b376-2e8eda058a7d","Type":"ContainerStarted","Data":"04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5"} Oct 30 07:52:09.050798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:09.050747 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" podStartSLOduration=2.050731203 podStartE2EDuration="2.050731203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:09.048990418 +0000 UTC m=+3292.739907855" watchObservedRunningTime="2026-10-30 07:52:09.050731203 +0000 UTC m=+3292.741648641" Oct 30 07:52:10.019103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.019066 5894 generic.go:356] "Generic (PLEG): container finished" podID="7cd6bedb-1158-401b-b376-2e8eda058a7d" containerID="e2477cceb712d69e92563f7990312bcfc94e8fadc1fe7b81c3ac59b2381561f0" exitCode=0 Oct 30 07:52:10.019558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.019149 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" event={"ID":"7cd6bedb-1158-401b-b376-2e8eda058a7d","Type":"ContainerDied","Data":"e2477cceb712d69e92563f7990312bcfc94e8fadc1fe7b81c3ac59b2381561f0"} Oct 30 07:52:10.112933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.112895 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:10.428077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.428051 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:52:10.471284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471255 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471325 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471446 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471476 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471582 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:10.471629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471623 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471676 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471712 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.471786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471737 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:10.471786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471759 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.472026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471851 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\") pod \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\" (UID: \"e16ed888-fd3a-4219-8440-d1ee0dc6e82a\") " Oct 30 07:52:10.472026 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.471955 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:10.472147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.472063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:10.472442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.472323 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474443 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474486 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474508 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474523 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474536 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.474758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.474550 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.476611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.476585 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "kube-api-access-c8rbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:10.476966 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.476940 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:52:10.477335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.477315 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy" pod "e16ed888-fd3a-4219-8440-d1ee0dc6e82a" (UID: "e16ed888-fd3a-4219-8440-d1ee0dc6e82a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:10.601405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.590967 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rbv\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-kube-api-access-c8rbv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.601405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.591011 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.601405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.591037 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:10.601405 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:10.591066 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e16ed888-fd3a-4219-8440-d1ee0dc6e82a-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:11.028770 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.028732 5894 generic.go:356] "Generic (PLEG): container finished" podID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerID="6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57" exitCode=0 Oct 30 07:52:11.029247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.028800 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" Oct 30 07:52:11.029247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.028819 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" event={"ID":"e16ed888-fd3a-4219-8440-d1ee0dc6e82a","Type":"ContainerDied","Data":"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57"} Oct 30 07:52:11.029247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.028901 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g" event={"ID":"e16ed888-fd3a-4219-8440-d1ee0dc6e82a","Type":"ContainerDied","Data":"660da3d4e1885838cb4d46bbea84a7018f5f2f884b12be6643ab0aeebff89798"} Oct 30 07:52:11.029247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.028923 5894 scope.go:122] "RemoveContainer" containerID="6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57" Oct 30 07:52:11.041352 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.041327 5894 scope.go:122] "RemoveContainer" containerID="6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57" Oct 30 07:52:11.041673 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:52:11.041620 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57\": container with ID starting with 6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57 not found: ID does not exist" containerID="6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57" Oct 30 07:52:11.041742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.041682 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57"} err="failed to get container status \"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57\": rpc error: code = NotFound desc = could not find container \"6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57\": container with ID starting with 6791f463cf67eba9251bf8b8eac1a1e13c91a053a4a326712a1ef25f2c7a3b57 not found: ID does not exist" Oct 30 07:52:11.085607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.085576 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:52:11.101681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.099916 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2f845b-openshift-default-5948cd7765-t9t2g"] Oct 30 07:52:11.186684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.186621 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:11.303566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.303484 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\") pod \"7cd6bedb-1158-401b-b376-2e8eda058a7d\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " Oct 30 07:52:11.303787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.303770 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\" (UniqueName: \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\") pod \"7cd6bedb-1158-401b-b376-2e8eda058a7d\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " Oct 30 07:52:11.303916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.303896 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\") pod \"7cd6bedb-1158-401b-b376-2e8eda058a7d\" (UID: \"7cd6bedb-1158-401b-b376-2e8eda058a7d\") " Oct 30 07:52:11.304250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.304227 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca" pod "7cd6bedb-1158-401b-b376-2e8eda058a7d" (UID: "7cd6bedb-1158-401b-b376-2e8eda058a7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:11.304550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.304530 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:11.305900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.305873 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca" pod "7cd6bedb-1158-401b-b376-2e8eda058a7d" (UID: "7cd6bedb-1158-401b-b376-2e8eda058a7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:52:11.305979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.305919 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn" pod "7cd6bedb-1158-401b-b376-2e8eda058a7d" (UID: "7cd6bedb-1158-401b-b376-2e8eda058a7d"). InnerVolumeSpecName "kube-api-access-bvjhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:11.406234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.406194 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cd6bedb-1158-401b-b376-2e8eda058a7d-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:11.406234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.406229 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvjhn\" (UniqueName: \"kubernetes.io/projected/7cd6bedb-1158-401b-b376-2e8eda058a7d-kube-api-access-bvjhn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:11.847853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:11.847819 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:12.040316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:12.040283 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" Oct 30 07:52:12.040316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:12.040304 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84" event={"ID":"7cd6bedb-1158-401b-b376-2e8eda058a7d","Type":"ContainerDied","Data":"04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5"} Oct 30 07:52:12.040785 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:12.040345 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04fde7dfd3e3ba934f97dc0ee2a051dd0aec02923fb30d689290ec2a7cdd04d5" Oct 30 07:52:12.435395 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:12.435356 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" path="/var/lib/kubelet/pods/e16ed888-fd3a-4219-8440-d1ee0dc6e82a/volumes" Oct 30 07:52:13.622267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:13.622228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:15.248002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:15.247961 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:16.943368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:16.943330 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:18.725449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.725409 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 07:52:18.728214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.728182 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" Oct 30 07:52:18.728214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.728212 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" Oct 30 07:52:18.728444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.728392 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7cd6bedb-1158-401b-b376-2e8eda058a7d" containerName="revoke-keys" Oct 30 07:52:18.728444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.728405 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cd6bedb-1158-401b-b376-2e8eda058a7d" containerName="revoke-keys" Oct 30 07:52:18.728925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.728904 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cd6bedb-1158-401b-b376-2e8eda058a7d" containerName="revoke-keys" Oct 30 07:52:18.729281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.729263 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16ed888-fd3a-4219-8440-d1ee0dc6e82a" containerName="istio-proxy" Oct 30 07:52:18.758243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.758130 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 07:52:18.758243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.758191 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:52:18.758574 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.758550 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:18.760598 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.760517 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:18.781122 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.781091 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-352fb3-serving-cert\"" Oct 30 07:52:18.922672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.922439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:18.955659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.946795 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:18.955659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.946853 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6pg\" (UniqueName: \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:18.955659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.946896 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:18.955659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:18.947119 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.049110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.048580 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.049110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.048790 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.049110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.048814 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6pg\" (UniqueName: \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.049110 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.048836 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.050455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.050200 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.053875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.053737 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.054760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.054388 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.077105 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.076464 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6pg\" (UniqueName: \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\") pod \"maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.116954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.116925 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:19.327430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.327334 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw"] Oct 30 07:52:19.334826 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.334796 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.352664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.351718 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:52:19.352975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.352939 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:52:19.353605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.353573 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw"] Oct 30 07:52:19.461677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.461609 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 07:52:19.462586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.462560 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.462745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.462606 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxc22\" (UniqueName: \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.462865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.462842 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.565127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.565086 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.565354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.565189 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxc22\" (UniqueName: \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.565433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.565353 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.566000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.565969 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.567754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.567733 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.589454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.589389 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxc22\" (UniqueName: \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\") pod \"maas-api-revoke-sub-9ce4318e-mvbtw\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:19.665116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:19.665080 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:20.083732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.083149 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw"] Oct 30 07:52:20.205792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.199360 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" event={"ID":"af913dfa-4d24-407d-8e22-cb03ead91b61","Type":"ContainerStarted","Data":"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a"} Oct 30 07:52:20.205792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.199753 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:20.205792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.199795 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" event={"ID":"af913dfa-4d24-407d-8e22-cb03ead91b61","Type":"ContainerStarted","Data":"d82ae97f3b8c71f7c648c15af894c9a7fe250187b8c5aeb2c500de21a4c94e93"} Oct 30 07:52:20.244065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.243986 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" podStartSLOduration=1.816194745 podStartE2EDuration="2.243970648s" totalImagesPullingTime="427.775903ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:20.241614845 +0000 UTC m=+3303.932532276" watchObservedRunningTime="2026-10-30 07:52:20.243970648 +0000 UTC m=+3303.934888093" Oct 30 07:52:20.735581 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:20.735546 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:21.210421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:21.210378 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" event={"ID":"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7","Type":"ContainerStarted","Data":"15f6d152f0f63c83d923330d254c9fb3170220245a4690b95adeb755ba2c5e9e"} Oct 30 07:52:21.210937 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:21.210433 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" event={"ID":"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7","Type":"ContainerStarted","Data":"829df19624f5f68aa4f552c8279a3611f5735b920a91226565e2ceaece90e787"} Oct 30 07:52:21.836617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:21.836584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:22.220935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.220850 5894 generic.go:356] "Generic (PLEG): container finished" podID="577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" containerID="15f6d152f0f63c83d923330d254c9fb3170220245a4690b95adeb755ba2c5e9e" exitCode=0 Oct 30 07:52:22.221338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.220932 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" event={"ID":"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7","Type":"ContainerDied","Data":"15f6d152f0f63c83d923330d254c9fb3170220245a4690b95adeb755ba2c5e9e"} Oct 30 07:52:22.410603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.410573 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:22.490844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.490750 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 07:52:22.491806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.491772 5894 cpu_manager.go:414] "containerMap: removing container" podUID="577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" containerName="revoke-keys" Oct 30 07:52:22.491806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.491794 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" containerName="revoke-keys" Oct 30 07:52:22.492036 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.491950 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" containerName="revoke-keys" Oct 30 07:52:22.495923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.495896 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.514542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.514502 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-268576-serving-cert\"" Oct 30 07:52:22.518783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.518752 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 07:52:22.529420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.529357 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\") pod \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " Oct 30 07:52:22.529578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.529498 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\") pod \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " Oct 30 07:52:22.529709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.529610 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\" (UniqueName: \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\") pod \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\" (UID: \"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7\") " Oct 30 07:52:22.531559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.531527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca" pod "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" (UID: "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:22.533012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.532984 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca" pod "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" (UID: "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:52:22.535136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.535109 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22" pod "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" (UID: "577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7"). InnerVolumeSpecName "kube-api-access-bxc22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:22.598298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.598258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:22.631623 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.631578 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.631869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.631843 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6f8\" (UniqueName: \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.631943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.631893 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.631996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.631982 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.632183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.632164 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:22.632231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.632188 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxc22\" (UniqueName: \"kubernetes.io/projected/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-kube-api-access-bxc22\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:22.632231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.632201 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:22.726660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.726148 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:22.738665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.734674 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6f8\" (UniqueName: \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.738665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.734728 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.738665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.734786 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.738665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.735079 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.738665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.736269 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.761708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.752400 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.761708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.754816 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.766693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.765984 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6f8\" (UniqueName: \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\") pod \"maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.825666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.825603 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:22.839604 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:22.839567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:23.142787 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:52:23.142755 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee13a129_5502_4364_9d45_2681daca4624.slice/crio-ed202fb94ef6b8efab678fe0988d90fc7b80a572cdf16c7e95cda752f2763d9b WatchSource:0}: Error finding container ed202fb94ef6b8efab678fe0988d90fc7b80a572cdf16c7e95cda752f2763d9b: Status 404 returned error can't find the container with id ed202fb94ef6b8efab678fe0988d90fc7b80a572cdf16c7e95cda752f2763d9b Oct 30 07:52:23.144759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:23.144730 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 07:52:23.231514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:23.231481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" event={"ID":"ee13a129-5502-4364-9d45-2681daca4624","Type":"ContainerStarted","Data":"ed202fb94ef6b8efab678fe0988d90fc7b80a572cdf16c7e95cda752f2763d9b"} Oct 30 07:52:23.233038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:23.233011 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" event={"ID":"577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7","Type":"ContainerDied","Data":"829df19624f5f68aa4f552c8279a3611f5735b920a91226565e2ceaece90e787"} Oct 30 07:52:23.233038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:23.233035 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw" Oct 30 07:52:23.233248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:23.233053 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="829df19624f5f68aa4f552c8279a3611f5735b920a91226565e2ceaece90e787" Oct 30 07:52:24.242525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:24.242484 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" event={"ID":"ee13a129-5502-4364-9d45-2681daca4624","Type":"ContainerStarted","Data":"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45"} Oct 30 07:52:24.242945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:24.242552 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:24.285827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:24.285776 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" podStartSLOduration=1.845579876 podStartE2EDuration="2.285762132s" totalImagesPullingTime="440.182256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:24.285115491 +0000 UTC m=+3307.976032928" watchObservedRunningTime="2026-10-30 07:52:24.285762132 +0000 UTC m=+3307.976679598" Oct 30 07:52:24.530930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:24.530896 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:26.217042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:26.217010 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 07:52:26.331792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:26.331748 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:28.044730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:28.044692 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:29.843329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:29.843293 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:30.258586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:30.258508 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 07:52:31.640837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:31.640802 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:31.942762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:31.942670 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:32.303557 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:32.303517 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:33.360014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:33.359974 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:33.800253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:33.800205 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" podUID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" containerName="main" containerID="cri-o://06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be" gracePeriod=45 Oct 30 07:52:34.107096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.107071 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:52:34.181483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181452 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\" (UniqueName: \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.181713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181505 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.181713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181609 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.181713 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181701 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.181908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181773 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.181972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181905 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:34.181972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.181945 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\") pod \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\" (UID: \"88afd2e9-56a1-4217-9d8c-7b3def0982d7\") " Oct 30 07:52:34.182761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.182246 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:34.182761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.182254 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:34.182761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.182697 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.182761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.182724 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.182761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.182737 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.184399 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.184372 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "kube-api-access-pstcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:34.184503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.184453 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:34.184731 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.184702 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs" pod "88afd2e9-56a1-4217-9d8c-7b3def0982d7" (UID: "88afd2e9-56a1-4217-9d8c-7b3def0982d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:52:34.284381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.284350 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88afd2e9-56a1-4217-9d8c-7b3def0982d7-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.284381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.284379 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88afd2e9-56a1-4217-9d8c-7b3def0982d7-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.284381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.284389 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstcf\" (UniqueName: \"kubernetes.io/projected/88afd2e9-56a1-4217-9d8c-7b3def0982d7-kube-api-access-pstcf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:34.335223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.335135 5894 generic.go:356] "Generic (PLEG): container finished" podID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" containerID="06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be" exitCode=0 Oct 30 07:52:34.335223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.335184 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" Oct 30 07:52:34.335223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.335210 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" event={"ID":"88afd2e9-56a1-4217-9d8c-7b3def0982d7","Type":"ContainerDied","Data":"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be"} Oct 30 07:52:34.335461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.335258 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t" event={"ID":"88afd2e9-56a1-4217-9d8c-7b3def0982d7","Type":"ContainerDied","Data":"c6bb694ddfc889c88c11f8eae4b99a2369a50344a4b650e7ea76ff84dd043c11"} Oct 30 07:52:34.335461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.335278 5894 scope.go:122] "RemoveContainer" containerID="06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be" Oct 30 07:52:34.348160 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.348139 5894 scope.go:122] "RemoveContainer" containerID="06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be" Oct 30 07:52:34.348497 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:52:34.348475 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be\": container with ID starting with 06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be not found: ID does not exist" containerID="06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be" Oct 30 07:52:34.348551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.348506 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be"} err="failed to get container status \"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be\": rpc error: code = NotFound desc = could not find container \"06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be\": container with ID starting with 06058af865f745051947a1a15284f85c9e3df1c015200c87315034e30a22d9be not found: ID does not exist" Oct 30 07:52:34.392348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.392309 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:52:34.396796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.396765 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d5f604/test-model-d5f604-kserve-7855cb9697-zsv6t"] Oct 30 07:52:34.435856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:34.435814 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" path="/var/lib/kubelet/pods/88afd2e9-56a1-4217-9d8c-7b3def0982d7/volumes" Oct 30 07:52:34.831489 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:52:34.831448 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:52:35.099094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:35.098999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:36.772216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:36.772176 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:38.491996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:38.491960 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:40.242789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:40.242745 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:41.778580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:41.778541 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:52:41.802369 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:41.802284 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:41.931752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:41.931577 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:42.193853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.193744 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p"] Oct 30 07:52:42.195179 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.195149 5894 cpu_manager.go:414] "containerMap: removing container" podUID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" containerName="main" Oct 30 07:52:42.195367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.195351 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" containerName="main" Oct 30 07:52:42.195790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.195772 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="88afd2e9-56a1-4217-9d8c-7b3def0982d7" containerName="main" Oct 30 07:52:42.202761 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.202738 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.216517 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.216472 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p"] Oct 30 07:52:42.219478 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.219456 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:52:42.219709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.219691 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:52:42.347244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.347205 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.347450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.347423 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.347540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.347529 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cwf\" (UniqueName: \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.448743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.448661 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.448919 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.448768 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cwf\" (UniqueName: \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.448971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.448914 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.449577 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.449547 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.451552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.451525 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.472515 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.472478 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cwf\" (UniqueName: \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\") pod \"maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.525086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.525047 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:42.846097 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:52:42.846066 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa08af85_ad0d_4e31_8934_ab750175bc96.slice/crio-be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1 WatchSource:0}: Error finding container be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1: Status 404 returned error can't find the container with id be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1 Oct 30 07:52:42.848277 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:42.848252 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p"] Oct 30 07:52:43.424710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:43.424669 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" event={"ID":"aa08af85-ad0d-4e31-8934-ab750175bc96","Type":"ContainerStarted","Data":"a7abaf2a7cc1cc57f4bcee4d46a7f227ff2ab7fd1734922441614e26996f1825"} Oct 30 07:52:43.424899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:43.424730 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" event={"ID":"aa08af85-ad0d-4e31-8934-ab750175bc96","Type":"ContainerStarted","Data":"be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1"} Oct 30 07:52:43.467660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:43.466731 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" podStartSLOduration=1.466711111 podStartE2EDuration="1.466711111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:52:43.46497237 +0000 UTC m=+3327.155889917" watchObservedRunningTime="2026-10-30 07:52:43.466711111 +0000 UTC m=+3327.157628550" Oct 30 07:52:44.079306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:44.079276 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:44.436607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:44.436560 5894 generic.go:356] "Generic (PLEG): container finished" podID="aa08af85-ad0d-4e31-8934-ab750175bc96" containerID="a7abaf2a7cc1cc57f4bcee4d46a7f227ff2ab7fd1734922441614e26996f1825" exitCode=0 Oct 30 07:52:44.436841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:44.436649 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" event={"ID":"aa08af85-ad0d-4e31-8934-ab750175bc96","Type":"ContainerDied","Data":"a7abaf2a7cc1cc57f4bcee4d46a7f227ff2ab7fd1734922441614e26996f1825"} Oct 30 07:52:45.631887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.631861 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:45.706697 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.706561 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\") pod \"aa08af85-ad0d-4e31-8934-ab750175bc96\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " Oct 30 07:52:45.706910 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.706889 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\") pod \"aa08af85-ad0d-4e31-8934-ab750175bc96\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " Oct 30 07:52:45.706992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.706978 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\" (UniqueName: \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\") pod \"aa08af85-ad0d-4e31-8934-ab750175bc96\" (UID: \"aa08af85-ad0d-4e31-8934-ab750175bc96\") " Oct 30 07:52:45.707412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.707379 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca" pod "aa08af85-ad0d-4e31-8934-ab750175bc96" (UID: "aa08af85-ad0d-4e31-8934-ab750175bc96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:52:45.707679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.707619 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:45.709096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.709065 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca" pod "aa08af85-ad0d-4e31-8934-ab750175bc96" (UID: "aa08af85-ad0d-4e31-8934-ab750175bc96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:52:45.709496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.709477 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf" pod "aa08af85-ad0d-4e31-8934-ab750175bc96" (UID: "aa08af85-ad0d-4e31-8934-ab750175bc96"). InnerVolumeSpecName "kube-api-access-j8cwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:45.808999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.808957 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8cwf\" (UniqueName: \"kubernetes.io/projected/aa08af85-ad0d-4e31-8934-ab750175bc96-kube-api-access-j8cwf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:45.808999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.808995 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa08af85-ad0d-4e31-8934-ab750175bc96-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:45.918450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:45.918412 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:46.455142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:46.455111 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" Oct 30 07:52:46.455305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:46.455144 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p" event={"ID":"aa08af85-ad0d-4e31-8934-ab750175bc96","Type":"ContainerDied","Data":"be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1"} Oct 30 07:52:46.455305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:46.455196 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be9549c39b694157ab9a8673880f6a97d01b68f696b7dfa2503ec68d24eb0fa1" Oct 30 07:52:47.783664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:47.783603 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:49.716438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:49.716393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:51.390056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:51.390018 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:52.657829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:52.657788 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:52:52.695416 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:52.695379 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:52:52.813121 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:52.813073 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:52.952805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:52.952614 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:53.117683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:53.117646 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:54.835998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:54.835960 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:56.579627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:56.579590 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:56.803957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:56.803867 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" podUID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" containerName="main" containerID="cri-o://bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882" gracePeriod=45 Oct 30 07:52:57.112936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.112908 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:52:57.141549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141512 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\" (UniqueName: \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.141772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141559 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.141772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141695 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.141920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141776 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.141920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141903 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.142043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.141941 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\") pod \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\" (UID: \"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058\") " Oct 30 07:52:57.142043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142000 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:57.142128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142053 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:57.142309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142284 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:57.142586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142558 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.142586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142581 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.142743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.142595 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.144119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.144093 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:52:57.144238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.144124 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "kube-api-access-f5sxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:52:57.144238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.144145 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs" pod "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" (UID: "1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:52:57.244319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.244231 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.244319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.244265 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5sxh\" (UniqueName: \"kubernetes.io/projected/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-kube-api-access-f5sxh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.244319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.244275 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:52:57.561880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.561842 5894 generic.go:356] "Generic (PLEG): container finished" podID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" containerID="bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882" exitCode=0 Oct 30 07:52:57.562099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.561890 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" Oct 30 07:52:57.562099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.561923 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" event={"ID":"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058","Type":"ContainerDied","Data":"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882"} Oct 30 07:52:57.562099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.561969 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754" event={"ID":"1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058","Type":"ContainerDied","Data":"0ab57c49c2f98c71766464a4889fb761eca543513d0624fec9420baa6f193e28"} Oct 30 07:52:57.562099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.561991 5894 scope.go:122] "RemoveContainer" containerID="bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882" Oct 30 07:52:57.574419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.574396 5894 scope.go:122] "RemoveContainer" containerID="bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882" Oct 30 07:52:57.574771 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:52:57.574749 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882\": container with ID starting with bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882 not found: ID does not exist" containerID="bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882" Oct 30 07:52:57.574838 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.574779 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882"} err="failed to get container status \"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882\": rpc error: code = NotFound desc = could not find container \"bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882\": container with ID starting with bb24a809408654fa389687106ab87c4f5cbd69e0a8b02955d0dbb6118b6be882 not found: ID does not exist" Oct 30 07:52:57.617304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.617270 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:52:57.621657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.621614 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-63f24b/test-model-63f24b-kserve-6448b94c6f-q4754"] Oct 30 07:52:57.659014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.658973 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" containerID="cri-o://292ad0b6d2c7d7e8904470c93fd81f3bc631b1ae77ef70e268a7ef1fbccfc474" gracePeriod=25 Oct 30 07:52:57.696231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:57.696187 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" containerID="cri-o://9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709" gracePeriod=25 Oct 30 07:52:58.410796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:58.410751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:58.438019 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:58.437983 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" path="/var/lib/kubelet/pods/1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058/volumes" Oct 30 07:52:58.756918 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:58.756816 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:52:58.804317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:58.804279 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:52:58.804493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:52:58.804282 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:53:00.163793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:00.163757 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:01.948463 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:01.948423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:03.565875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:03.565842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:04.834060 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:53:04.834021 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:53:05.416098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:05.416043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:07.286663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:07.285923 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:53:07.304480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:07.304444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:08.803909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:08.803866 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:53:08.804432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:08.803993 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:53:09.275714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:09.270825 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:11.385285 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:11.385249 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:13.332343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:13.332247 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:15.140993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:15.140958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:17.078782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:17.078740 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:17.779900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:17.779870 5894 generic.go:356] "Generic (PLEG): container finished" podID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerID="292ad0b6d2c7d7e8904470c93fd81f3bc631b1ae77ef70e268a7ef1fbccfc474" exitCode=0 Oct 30 07:53:17.780055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:17.779953 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" event={"ID":"e15ae5eb-3dd9-4546-9e70-01fb89c57e51","Type":"ContainerDied","Data":"292ad0b6d2c7d7e8904470c93fd81f3bc631b1ae77ef70e268a7ef1fbccfc474"} Oct 30 07:53:18.023253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.023223 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:53:18.050932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.050905 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:53:18.072292 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.072251 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\" (UniqueName: \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\") pod \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " Oct 30 07:53:18.072472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.072399 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\" (UniqueName: \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\") pod \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " Oct 30 07:53:18.072472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.072424 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\") pod \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\" (UID: \"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900\") " Oct 30 07:53:18.072734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.072707 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\") pod \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\" (UID: \"e15ae5eb-3dd9-4546-9e70-01fb89c57e51\") " Oct 30 07:53:18.072892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.072872 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume" pod "d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" (UID: "d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:18.073328 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.073293 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume" pod "e15ae5eb-3dd9-4546-9e70-01fb89c57e51" (UID: "e15ae5eb-3dd9-4546-9e70-01fb89c57e51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:18.073499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.073474 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:18.073499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.073501 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:18.075531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.075509 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq" pod "e15ae5eb-3dd9-4546-9e70-01fb89c57e51" (UID: "e15ae5eb-3dd9-4546-9e70-01fb89c57e51"). InnerVolumeSpecName "kube-api-access-nw4nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:18.075705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.075679 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf" pod "d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" (UID: "d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900"). InnerVolumeSpecName "kube-api-access-9wnsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:18.175803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.175761 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnsf\" (UniqueName: \"kubernetes.io/projected/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900-kube-api-access-9wnsf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:18.175803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.175804 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw4nq\" (UniqueName: \"kubernetes.io/projected/e15ae5eb-3dd9-4546-9e70-01fb89c57e51-kube-api-access-nw4nq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:18.790205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.790145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" event={"ID":"e15ae5eb-3dd9-4546-9e70-01fb89c57e51","Type":"ContainerDied","Data":"348970bc9baea84d7a1b1ed11d5bbf8f85312e8395f2aa499e49c75b8c49723c"} Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.790255 5894 scope.go:122] "RemoveContainer" containerID="292ad0b6d2c7d7e8904470c93fd81f3bc631b1ae77ef70e268a7ef1fbccfc474" Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.790274 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4" Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.790313 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.791799 5894 generic.go:356] "Generic (PLEG): container finished" podID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerID="9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709" exitCode=0 Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.791856 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" event={"ID":"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900","Type":"ContainerDied","Data":"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709"} Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.791865 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.791892 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd" event={"ID":"d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900","Type":"ContainerDied","Data":"2be6a51913db02d77ac457db1e21ca92d6876c10c05e26d95f9781897ba7b6f9"} Oct 30 07:53:18.838186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.820276 5894 scope.go:122] "RemoveContainer" containerID="9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709" Oct 30 07:53:18.863867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.863758 5894 scope.go:122] "RemoveContainer" containerID="9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709" Oct 30 07:53:18.872335 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:53:18.872283 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709\": container with ID starting with 9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709 not found: ID does not exist" containerID="9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709" Oct 30 07:53:18.872335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.872345 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709"} err="failed to get container status \"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709\": rpc error: code = NotFound desc = could not find container \"9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709\": container with ID starting with 9f6084c4dea1bd4acff11fbc53305886306e0cc92c9d8786f8309b720fe79709 not found: ID does not exist" Oct 30 07:53:18.876499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.876466 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:53:18.881379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.881351 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5f604-6fcdc9fb7-lgvdd"] Oct 30 07:53:18.921347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.921304 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:53:18.928996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:18.928963 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d5f604-59df4bc678-xw7x4"] Oct 30 07:53:19.806282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.806241 5894 generic.go:356] "Generic (PLEG): container finished" podID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerID="c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5" exitCode=0 Oct 30 07:53:19.807094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.806323 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerDied","Data":"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5"} Oct 30 07:53:19.807094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.806840 5894 scope.go:122] "RemoveContainer" containerID="c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5" Oct 30 07:53:19.808976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.808942 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:53:19.809104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.809031 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:53:19.809104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:19.809083 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:53:20.435007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:20.434970 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" path="/var/lib/kubelet/pods/d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900/volumes" Oct 30 07:53:20.435454 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:20.435440 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" path="/var/lib/kubelet/pods/e15ae5eb-3dd9-4546-9e70-01fb89c57e51/volumes" Oct 30 07:53:20.541844 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:20.541811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:20.819857 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:20.819819 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerStarted","Data":"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9"} Oct 30 07:53:21.291659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:21.277996 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:53:21.296664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:21.295848 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:21.296664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:21.295915 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:21.456059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:21.456026 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:53:22.293656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:22.293110 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:22.293656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:22.293191 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:22.494660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:22.493775 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:23.032846 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:23.032807 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248"] Oct 30 07:53:23.045063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:23.045028 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da45bd21-sh248"] Oct 30 07:53:23.292716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:23.286038 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:23.292716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:23.286111 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:24.032384 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.032348 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx"] Oct 30 07:53:24.044471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.044432 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f68e69d7-tpvjx"] Oct 30 07:53:24.222178 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.222140 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:24.284709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.282775 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:24.284709 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.282855 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:24.434807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.434770 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3521a0ac-248a-492f-bfc4-26d5a624f260" path="/var/lib/kubelet/pods/3521a0ac-248a-492f-bfc4-26d5a624f260/volumes" Oct 30 07:53:24.435320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:24.435302 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7fd062e-b1f8-4c27-8d4e-25e431393aa2" path="/var/lib/kubelet/pods/c7fd062e-b1f8-4c27-8d4e-25e431393aa2/volumes" Oct 30 07:53:25.280075 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:25.280038 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:25.280733 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:25.280702 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:26.117315 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:26.109440 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:26.281998 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:26.281962 5894 patch_prober.go:29] interesting pod/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" start-of-body= Oct 30 07:53:26.282597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:26.282566 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.85:15021/healthz/ready\": dial tcp 10.134.1.85:15021: connect: connection refused" Oct 30 07:53:27.283182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:27.283143 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:53:27.284335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:27.284308 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 07:53:27.949930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:27.949889 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:28.963306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:28.963264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:29.786535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:29.786439 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:30.742793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.742756 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:53:30.744223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744195 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" containerName="main" Oct 30 07:53:30.744223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744222 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" containerName="main" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744252 5894 cpu_manager.go:414] "containerMap: removing container" podUID="aa08af85-ad0d-4e31-8934-ab750175bc96" containerName="revoke-keys" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744262 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa08af85-ad0d-4e31-8934-ab750175bc96" containerName="revoke-keys" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744280 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744287 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744305 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744313 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744546 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bfaf8fe-add8-4dac-83ac-ad7d2fe2d058" containerName="main" Oct 30 07:53:30.744565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744570 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa08af85-ad0d-4e31-8934-ab750175bc96" containerName="revoke-keys" Oct 30 07:53:30.745057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744582 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e15ae5eb-3dd9-4546-9e70-01fb89c57e51" containerName="payload-processing" Oct 30 07:53:30.745057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.744598 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e0bbd8-5c3f-48ad-abdb-7fbfe47cb900" containerName="payload-pre-processing" Oct 30 07:53:30.749403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.749341 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.767954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.767923 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:53:30.775884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.775850 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-057cfa\"/\"openshift-service-ca.crt\"" Oct 30 07:53:30.776064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.775896 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-057cfa\"/\"kube-root-ca.crt\"" Oct 30 07:53:30.787449 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.787426 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-057cfa\"/\"default-dockercfg-zds7n\"" Oct 30 07:53:30.787597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.787557 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-057cfa\"/\"test-model-057cfa-kserve-self-signed-certs\"" Oct 30 07:53:30.811090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.811052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:30.882798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.882761 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.883004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.882907 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bbjs\" (UniqueName: \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.883060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.883021 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.883060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.883052 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.883175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.883087 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.883281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.883263 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.984743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.984705 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.985068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.985042 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.985359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.985327 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.985619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.985598 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.985863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.985843 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bbjs\" (UniqueName: \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.986180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.986157 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.987957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.987931 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.990027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.988675 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:30.990163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:30.989221 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:31.001386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.001312 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:31.003535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.003491 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:31.033389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.033353 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bbjs\" (UniqueName: \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\") pod \"test-model-057cfa-kserve-5b786f8d6c-npfgf\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:31.069520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.069486 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:31.217603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.208538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:31.559389 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:53:31.559357 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f6e4b0e_cbeb_40b1_95cb_039608321116.slice/crio-b81bc21df1480e85f8290c3ee3eca4efc8a8c69860b7d310a6ba4a4a7726a11f WatchSource:0}: Error finding container b81bc21df1480e85f8290c3ee3eca4efc8a8c69860b7d310a6ba4a4a7726a11f: Status 404 returned error can't find the container with id b81bc21df1480e85f8290c3ee3eca4efc8a8c69860b7d310a6ba4a4a7726a11f Oct 30 07:53:31.561044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.561020 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:53:31.921707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:31.918933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:32.035271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.035222 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" event={"ID":"4f6e4b0e-cbeb-40b1-95cb-039608321116","Type":"ContainerStarted","Data":"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e"} Oct 30 07:53:32.035489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.035279 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" event={"ID":"4f6e4b0e-cbeb-40b1-95cb-039608321116","Type":"ContainerStarted","Data":"b81bc21df1480e85f8290c3ee3eca4efc8a8c69860b7d310a6ba4a4a7726a11f"} Oct 30 07:53:32.070547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.070516 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:32.105227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.105178 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" podStartSLOduration=2.105162904 podStartE2EDuration="2.105162904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:32.104795303 +0000 UTC m=+3375.795712740" watchObservedRunningTime="2026-10-30 07:53:32.105162904 +0000 UTC m=+3375.796080377" Oct 30 07:53:32.144571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.144529 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:32.462855 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:32.462823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:33.772656 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:33.770122 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:34.045605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.045570 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf"] Oct 30 07:53:34.057710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.057669 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb"] Oct 30 07:53:34.061842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.061820 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b35c51c-kvntf"] Oct 30 07:53:34.065551 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.065527 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f756628-2f2rb"] Oct 30 07:53:34.435540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.435459 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148b79ae-0821-42cf-b1e6-bf1aed3bff76" path="/var/lib/kubelet/pods/148b79ae-0821-42cf-b1e6-bf1aed3bff76/volumes" Oct 30 07:53:34.436104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.436087 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea9f335-2f30-4d45-8bc0-6c066a67b947" path="/var/lib/kubelet/pods/dea9f335-2f30-4d45-8bc0-6c066a67b947/volumes" Oct 30 07:53:34.838089 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:53:34.838034 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:53:34.857944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.857905 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr"] Oct 30 07:53:34.863311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.863284 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.882615 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.882584 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr"] Oct 30 07:53:34.885350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.885311 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.885480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.885372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-254hq\" (UniqueName: \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.885708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.885679 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.889452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.889409 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:53:34.889722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.889549 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:53:34.987379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.987332 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.987606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.987396 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-254hq\" (UniqueName: \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.987606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.987579 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.988238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.988213 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:34.989892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:34.989867 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:35.014927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:35.014895 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-254hq\" (UniqueName: \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\") pod \"maas-api-revoke-sub-ec867198-f8qdr\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:35.182537 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:35.182449 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:35.521372 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:53:35.521339 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c65fd04_901d_4f6b_88da_e4768d6801b9.slice/crio-37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39 WatchSource:0}: Error finding container 37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39: Status 404 returned error can't find the container with id 37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39 Oct 30 07:53:35.524603 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:35.524576 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr"] Oct 30 07:53:35.938140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:35.938099 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:36.074812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:36.074721 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" event={"ID":"7c65fd04-901d-4f6b-88da-e4768d6801b9","Type":"ContainerStarted","Data":"85349d363f5f144cf526eead54cd5a2f0701bab36bbdc8736617fe0312a3b1e1"} Oct 30 07:53:36.074812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:36.074790 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" event={"ID":"7c65fd04-901d-4f6b-88da-e4768d6801b9","Type":"ContainerStarted","Data":"37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39"} Oct 30 07:53:37.086891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:37.086856 5894 generic.go:356] "Generic (PLEG): container finished" podID="7c65fd04-901d-4f6b-88da-e4768d6801b9" containerID="85349d363f5f144cf526eead54cd5a2f0701bab36bbdc8736617fe0312a3b1e1" exitCode=0 Oct 30 07:53:37.087309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:37.086948 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" event={"ID":"7c65fd04-901d-4f6b-88da-e4768d6801b9","Type":"ContainerDied","Data":"85349d363f5f144cf526eead54cd5a2f0701bab36bbdc8736617fe0312a3b1e1"} Oct 30 07:53:37.717723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:37.717604 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:38.290431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.290402 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:38.342117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.342082 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\" (UniqueName: \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\") pod \"7c65fd04-901d-4f6b-88da-e4768d6801b9\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " Oct 30 07:53:38.342317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.342289 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\") pod \"7c65fd04-901d-4f6b-88da-e4768d6801b9\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " Oct 30 07:53:38.342404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.342350 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\") pod \"7c65fd04-901d-4f6b-88da-e4768d6801b9\" (UID: \"7c65fd04-901d-4f6b-88da-e4768d6801b9\") " Oct 30 07:53:38.342752 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.342722 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca" pod "7c65fd04-901d-4f6b-88da-e4768d6801b9" (UID: "7c65fd04-901d-4f6b-88da-e4768d6801b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:38.343087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.343063 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:38.344945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.344885 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq" pod "7c65fd04-901d-4f6b-88da-e4768d6801b9" (UID: "7c65fd04-901d-4f6b-88da-e4768d6801b9"). InnerVolumeSpecName "kube-api-access-254hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:38.344945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.344927 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca" pod "7c65fd04-901d-4f6b-88da-e4768d6801b9" (UID: "7c65fd04-901d-4f6b-88da-e4768d6801b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:53:38.444458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.444423 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c65fd04-901d-4f6b-88da-e4768d6801b9-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:38.444458 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:38.444457 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-254hq\" (UniqueName: \"kubernetes.io/projected/7c65fd04-901d-4f6b-88da-e4768d6801b9-kube-api-access-254hq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:39.105347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:39.105308 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" event={"ID":"7c65fd04-901d-4f6b-88da-e4768d6801b9","Type":"ContainerDied","Data":"37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39"} Oct 30 07:53:39.105548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:39.105366 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37b6068ed1f8905f3484570c6db8f56aeac333ba0c40d686d2d2057b2d15bd39" Oct 30 07:53:39.105548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:39.105328 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr" Oct 30 07:53:39.306836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:39.306803 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:39.963077 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:39.963041 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:40.232489 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:40.232403 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p"] Oct 30 07:53:40.243116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:40.243084 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d5f604-ctx5p"] Oct 30 07:53:40.245358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:40.245330 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:40.434745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:40.434710 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa08af85-ad0d-4e31-8934-ab750175bc96" path="/var/lib/kubelet/pods/aa08af85-ad0d-4e31-8934-ab750175bc96/volumes" Oct 30 07:53:40.669689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:40.669621 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:41.071193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:41.071158 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:41.084382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:41.084353 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:41.197282 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:41.197247 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:53:41.820293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:41.820256 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:42.207205 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:42.207125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:44.156307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:44.156264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:45.927221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:45.927184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:47.033055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.033023 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v"] Oct 30 07:53:47.045250 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.045219 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1c48a1-m627v"] Oct 30 07:53:47.339608 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.339515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:47.495183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.495136 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:47.658460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.658364 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:53:47.658679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.658593 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" containerID="cri-o://9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318" gracePeriod=30 Oct 30 07:53:47.679653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.677835 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:47.830707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:47.827894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:48.252210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.252171 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5"] Oct 30 07:53:48.253113 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.253093 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7c65fd04-901d-4f6b-88da-e4768d6801b9" containerName="revoke-keys" Oct 30 07:53:48.253170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.253114 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c65fd04-901d-4f6b-88da-e4768d6801b9" containerName="revoke-keys" Oct 30 07:53:48.253318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.253308 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c65fd04-901d-4f6b-88da-e4768d6801b9" containerName="revoke-keys" Oct 30 07:53:48.256281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.256262 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.283451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.283415 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5"] Oct 30 07:53:48.344979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.344950 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:53:48.345193 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.345043 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:53:48.426269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.426238 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pcl4\" (UniqueName: \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.426490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.426466 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.426540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.426514 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.434906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.434876 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d96c5296-77ef-4424-aa09-e83dcf885329" path="/var/lib/kubelet/pods/d96c5296-77ef-4424-aa09-e83dcf885329/volumes" Oct 30 07:53:48.528303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.528274 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.528506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.528358 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.528590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.528545 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pcl4\" (UniqueName: \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.529404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.529377 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.531065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.531041 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.551561 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.551528 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pcl4\" (UniqueName: \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\") pod \"maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.570512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.570477 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:48.806387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.806304 5894 patch_prober.go:29] interesting pod/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:53:48.806387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.806378 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:53:48.881774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:48.881743 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5"] Oct 30 07:53:48.882284 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:53:48.882259 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adefa29_ed2c_4bc7_b9e7_b4643a31904b.slice/crio-55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a WatchSource:0}: Error finding container 55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a: Status 404 returned error can't find the container with id 55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a Oct 30 07:53:49.035135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.035098 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd"] Oct 30 07:53:49.039218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.039193 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21e2467f-st8sd"] Oct 30 07:53:49.295676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.295616 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" event={"ID":"3adefa29-ed2c-4bc7-b9e7-b4643a31904b","Type":"ContainerStarted","Data":"fea161c70814f11856a7bb1fbb628316693e6acc703a83d0f5fb894268c96f32"} Oct 30 07:53:49.296062 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.295693 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" event={"ID":"3adefa29-ed2c-4bc7-b9e7-b4643a31904b","Type":"ContainerStarted","Data":"55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a"} Oct 30 07:53:49.338432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.338379 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" podStartSLOduration=1.338367122 podStartE2EDuration="1.338367122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:53:49.337964681 +0000 UTC m=+3393.028882118" watchObservedRunningTime="2026-10-30 07:53:49.338367122 +0000 UTC m=+3393.029284559" Oct 30 07:53:49.581070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:49.581033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:50.305873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:50.305844 5894 generic.go:356] "Generic (PLEG): container finished" podID="3adefa29-ed2c-4bc7-b9e7-b4643a31904b" containerID="fea161c70814f11856a7bb1fbb628316693e6acc703a83d0f5fb894268c96f32" exitCode=0 Oct 30 07:53:50.306313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:50.305937 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" event={"ID":"3adefa29-ed2c-4bc7-b9e7-b4643a31904b","Type":"ContainerDied","Data":"fea161c70814f11856a7bb1fbb628316693e6acc703a83d0f5fb894268c96f32"} Oct 30 07:53:50.434653 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:50.434605 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ca3eea-75d9-455f-9bd8-abf1f9e15868" path="/var/lib/kubelet/pods/19ca3eea-75d9-455f-9bd8-abf1f9e15868/volumes" Oct 30 07:53:51.340959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.340920 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:51.564435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.564407 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:51.622422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.622338 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\") pod \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " Oct 30 07:53:51.622578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.622514 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\" (UniqueName: \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\") pod \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " Oct 30 07:53:51.622722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.622702 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\") pod \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\" (UID: \"3adefa29-ed2c-4bc7-b9e7-b4643a31904b\") " Oct 30 07:53:51.623117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.623084 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca" pod "3adefa29-ed2c-4bc7-b9e7-b4643a31904b" (UID: "3adefa29-ed2c-4bc7-b9e7-b4643a31904b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:51.623548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.623525 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:51.625035 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.625005 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4" pod "3adefa29-ed2c-4bc7-b9e7-b4643a31904b" (UID: "3adefa29-ed2c-4bc7-b9e7-b4643a31904b"). InnerVolumeSpecName "kube-api-access-9pcl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:51.625438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.625420 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca" pod "3adefa29-ed2c-4bc7-b9e7-b4643a31904b" (UID: "3adefa29-ed2c-4bc7-b9e7-b4643a31904b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:53:51.725226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.725189 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pcl4\" (UniqueName: \"kubernetes.io/projected/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-kube-api-access-9pcl4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:51.725226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:51.725224 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3adefa29-ed2c-4bc7-b9e7-b4643a31904b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:52.339614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:52.339578 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" Oct 30 07:53:52.339833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:52.339579 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5" event={"ID":"3adefa29-ed2c-4bc7-b9e7-b4643a31904b","Type":"ContainerDied","Data":"55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a"} Oct 30 07:53:52.339833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:52.339722 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55833c695858de35bf90e081163b2a71af8ae90dcb9269711aa84f2e6f47223a" Oct 30 07:53:52.963965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:52.963932 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:53:53.054107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054078 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054188 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054229 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054322 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054406 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054441 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054499 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054743 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.054951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.054814 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\") pod \"a2474bd6-f0df-4748-9059-5510490cbcce\" (UID: \"a2474bd6-f0df-4748-9059-5510490cbcce\") " Oct 30 07:53:53.055992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.055947 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.056320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.056255 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.056320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.056287 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.056540 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.056519 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.056745 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.056714 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:53.060233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.060206 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "kube-api-access-5tdgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.060829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.060769 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:53:53.060829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.060807 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:53.061867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.061834 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy" pod "a2474bd6-f0df-4748-9059-5510490cbcce" (UID: "a2474bd6-f0df-4748-9059-5510490cbcce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:53:53.157811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157777 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.157811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157810 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157822 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2474bd6-f0df-4748-9059-5510490cbcce-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157834 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdgj\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-kube-api-access-5tdgj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157846 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157859 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157871 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2474bd6-f0df-4748-9059-5510490cbcce-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157881 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2474bd6-f0df-4748-9059-5510490cbcce-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.158046 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.157892 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2474bd6-f0df-4748-9059-5510490cbcce-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:53.200876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.200792 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:53.350374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.350340 5894 generic.go:356] "Generic (PLEG): container finished" podID="a2474bd6-f0df-4748-9059-5510490cbcce" containerID="9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318" exitCode=0 Oct 30 07:53:53.350607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.350399 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" Oct 30 07:53:53.350607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.350437 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" event={"ID":"a2474bd6-f0df-4748-9059-5510490cbcce","Type":"ContainerDied","Data":"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318"} Oct 30 07:53:53.350607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.350497 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz" event={"ID":"a2474bd6-f0df-4748-9059-5510490cbcce","Type":"ContainerDied","Data":"fc2d08103c83fa40d1e3b7ea5fe3e95f20013629fb60c11b8bd3a528da1054ea"} Oct 30 07:53:53.350607 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.350521 5894 scope.go:122] "RemoveContainer" containerID="9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318" Oct 30 07:53:53.363762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.363723 5894 scope.go:122] "RemoveContainer" containerID="9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318" Oct 30 07:53:53.364086 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:53:53.364064 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318\": container with ID starting with 9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318 not found: ID does not exist" containerID="9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318" Oct 30 07:53:53.364185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.364092 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318"} err="failed to get container status \"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318\": rpc error: code = NotFound desc = could not find container \"9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318\": container with ID starting with 9f65660ff0fb7a0e2d5638b4479259479e9d94eb39029164fffce4a732978318 not found: ID does not exist" Oct 30 07:53:53.406509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.406469 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:53:53.419241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:53.419211 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d5f604-openshift-default-5ccc6b8b6f-k4vzz"] Oct 30 07:53:54.435986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:54.435950 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" path="/var/lib/kubelet/pods/a2474bd6-f0df-4748-9059-5510490cbcce/volumes" Oct 30 07:53:54.807269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:54.802099 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:55.262010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.261923 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:53:55.285246 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.285206 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:55.787076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.787045 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656"] Oct 30 07:53:55.788052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788031 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" Oct 30 07:53:55.788052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788052 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" Oct 30 07:53:55.788163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788110 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3adefa29-ed2c-4bc7-b9e7-b4643a31904b" containerName="revoke-keys" Oct 30 07:53:55.788163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788115 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adefa29-ed2c-4bc7-b9e7-b4643a31904b" containerName="revoke-keys" Oct 30 07:53:55.788241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788231 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2474bd6-f0df-4748-9059-5510490cbcce" containerName="istio-proxy" Oct 30 07:53:55.788283 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.788251 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adefa29-ed2c-4bc7-b9e7-b4643a31904b" containerName="revoke-keys" Oct 30 07:53:55.792192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.792169 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:55.808819 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.808788 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:53:55.814865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.814835 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656"] Oct 30 07:53:55.822848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.822821 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:53:55.911825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.911788 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5xf\" (UniqueName: \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:55.912003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.911930 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:55.912184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:55.912164 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.014313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.014266 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.014553 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.014531 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.014715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.014695 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5xf\" (UniqueName: \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.015309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.015280 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.017837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.017804 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.041096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.041030 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5xf\" (UniqueName: \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\") pod \"maas-api-revoke-sub-113c4f1a-pv656\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.107314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.107279 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:56.344366 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.344260 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:56.426164 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:53:56.426133 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7800d6_4988_4410_b320_4b3871eefe57.slice/crio-d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad WatchSource:0}: Error finding container d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad: Status 404 returned error can't find the container with id d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad Oct 30 07:53:56.439701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.439667 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656"] Oct 30 07:53:56.561681 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:56.561612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:57.418318 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:57.418282 5894 generic.go:356] "Generic (PLEG): container finished" podID="6b7800d6-4988-4410-b320-4b3871eefe57" containerID="64e5508623f18a9587a86d42694dbbc9b52c6ce2b2ee8c3416ac8dcb38466bcc" exitCode=0 Oct 30 07:53:57.418793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:57.418361 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" event={"ID":"6b7800d6-4988-4410-b320-4b3871eefe57","Type":"ContainerDied","Data":"64e5508623f18a9587a86d42694dbbc9b52c6ce2b2ee8c3416ac8dcb38466bcc"} Oct 30 07:53:57.418793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:57.418415 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" event={"ID":"6b7800d6-4988-4410-b320-4b3871eefe57","Type":"ContainerStarted","Data":"d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad"} Oct 30 07:53:58.034899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.034864 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j"] Oct 30 07:53:58.046875 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.046842 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e6c9cf0-6z97j"] Oct 30 07:53:58.396711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.395473 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:58.461711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.461680 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec58f03-272f-44eb-88a4-6fef1d1fc9d8" path="/var/lib/kubelet/pods/dec58f03-272f-44eb-88a4-6fef1d1fc9d8/volumes" Oct 30 07:53:58.463227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.463201 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:53:58.469063 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.468930 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:53:58.469247 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.469019 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.486967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.486848 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-352fb3\"/\"kube-root-ca.crt\"" Oct 30 07:53:58.487154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.487020 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-352fb3\"/\"default-dockercfg-qndjj\"" Oct 30 07:53:58.487154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.487121 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-352fb3\"/\"test-model-352fb3-kserve-self-signed-certs\"" Oct 30 07:53:58.487309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.487251 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-352fb3\"/\"openshift-service-ca.crt\"" Oct 30 07:53:58.542143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.542107 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.680936 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.681060 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.681224 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.681372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.681425 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.711140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.681450 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jcrb\" (UniqueName: \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.792356 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.792317 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.793372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.793349 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.794393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.794371 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.817771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.815877 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.818269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.818232 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.818575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.818542 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.818772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.818742 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jcrb\" (UniqueName: \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.818986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.818969 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.819116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.818673 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.819815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.819778 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.827672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.825352 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.854977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.851918 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jcrb\" (UniqueName: \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\") pod \"test-model-352fb3-kserve-677bd8bfd9-lzrvb\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:58.870716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:58.868327 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:59.022150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.022076 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\" (UniqueName: \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\") pod \"6b7800d6-4988-4410-b320-4b3871eefe57\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " Oct 30 07:53:59.022310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.022209 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\") pod \"6b7800d6-4988-4410-b320-4b3871eefe57\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " Oct 30 07:53:59.022310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.022268 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\") pod \"6b7800d6-4988-4410-b320-4b3871eefe57\" (UID: \"6b7800d6-4988-4410-b320-4b3871eefe57\") " Oct 30 07:53:59.022677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.022613 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca" pod "6b7800d6-4988-4410-b320-4b3871eefe57" (UID: "6b7800d6-4988-4410-b320-4b3871eefe57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:53:59.022941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.022923 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:59.024380 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.024358 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca" pod "6b7800d6-4988-4410-b320-4b3871eefe57" (UID: "6b7800d6-4988-4410-b320-4b3871eefe57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:53:59.024504 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.024449 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf" pod "6b7800d6-4988-4410-b320-4b3871eefe57" (UID: "6b7800d6-4988-4410-b320-4b3871eefe57"). InnerVolumeSpecName "kube-api-access-jr5xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:53:59.089668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.089601 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:53:59.125010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.124972 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr5xf\" (UniqueName: \"kubernetes.io/projected/6b7800d6-4988-4410-b320-4b3871eefe57-kube-api-access-jr5xf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:59.125010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.125005 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b7800d6-4988-4410-b320-4b3871eefe57-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:53:59.402549 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:53:59.402517 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2438141b_a7d8_4ea7_b0fe_55788b0713b1.slice/crio-4adff0e9d44143535ab6eb7c555d9c9f9febaa5243ce73f00205ed7d8a771d57 WatchSource:0}: Error finding container 4adff0e9d44143535ab6eb7c555d9c9f9febaa5243ce73f00205ed7d8a771d57: Status 404 returned error can't find the container with id 4adff0e9d44143535ab6eb7c555d9c9f9febaa5243ce73f00205ed7d8a771d57 Oct 30 07:53:59.403502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.403478 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:53:59.456445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.456413 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:53:59.478239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.478189 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" event={"ID":"2438141b-a7d8-4ea7-b0fe-55788b0713b1","Type":"ContainerStarted","Data":"4adff0e9d44143535ab6eb7c555d9c9f9febaa5243ce73f00205ed7d8a771d57"} Oct 30 07:53:59.480588 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.480557 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" event={"ID":"6b7800d6-4988-4410-b320-4b3871eefe57","Type":"ContainerDied","Data":"d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad"} Oct 30 07:53:59.480759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.480610 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d999763b24804a8550ea2f8e509945cfa6f33b6aecf603c66140d7eb963af9ad" Oct 30 07:53:59.480759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.480614 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656" Oct 30 07:53:59.489180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.489156 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:53:59.619119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.619088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:53:59.739396 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:53:59.739357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:00.034145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.034113 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8"] Oct 30 07:54:00.038303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.038265 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af70d619-h4pn8"] Oct 30 07:54:00.048030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.048002 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:00.435381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.435306 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55b0b65-9860-4d89-af4d-ebc585f254e8" path="/var/lib/kubelet/pods/a55b0b65-9860-4d89-af4d-ebc585f254e8/volumes" Oct 30 07:54:00.493711 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.493625 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" event={"ID":"2438141b-a7d8-4ea7-b0fe-55788b0713b1","Type":"ContainerStarted","Data":"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa"} Oct 30 07:54:00.541538 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:00.541477 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" podStartSLOduration=2.541461602 podStartE2EDuration="2.541461602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:00.538714634 +0000 UTC m=+3404.229632082" watchObservedRunningTime="2026-10-30 07:54:00.541461602 +0000 UTC m=+3404.232379039" Oct 30 07:54:01.090576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:01.090534 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:54:01.474303 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:01.474207 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:54:01.873194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:01.873158 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:03.712322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:03.712287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:04.457076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:04.457026 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" containerID="cri-o://4165841bfc3fc359b074cd7c32fd8686aa74c1ca5c7cdff249b0afe285caa958" gracePeriod=25 Oct 30 07:54:04.490029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:04.489968 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" containerID="cri-o://c8379124a295e72b929eb8ae0738d84308282d7ba6f87d6c655b3c50f8c66b5a" gracePeriod=25 Oct 30 07:54:04.608320 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:04.608283 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:05.015919 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:05.015836 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:54:05.525712 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:05.525596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:05.877317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:05.877278 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:05.877514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:05.877278 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:07.282275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:07.282240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:08.512995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:08.512953 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:09.090335 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:09.090301 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:54:09.103647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:09.103599 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:54:09.115524 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:09.115488 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:09.637257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:09.637220 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:54:10.288725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.288686 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" podUID="7769b684-807f-49b6-9231-475bc872e040" containerName="main" containerID="cri-o://a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68" gracePeriod=45 Oct 30 07:54:10.596045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.596017 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:54:10.637823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.637782 5894 generic.go:356] "Generic (PLEG): container finished" podID="7769b684-807f-49b6-9231-475bc872e040" containerID="a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68" exitCode=0 Oct 30 07:54:10.637823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.637825 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" Oct 30 07:54:10.638381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.637866 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" event={"ID":"7769b684-807f-49b6-9231-475bc872e040","Type":"ContainerDied","Data":"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68"} Oct 30 07:54:10.638381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.637913 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh" event={"ID":"7769b684-807f-49b6-9231-475bc872e040","Type":"ContainerDied","Data":"27cde8c3c513b1d22838822a0de5e972a7c6845f3f35cacf0b1b02da56145a3c"} Oct 30 07:54:10.638381 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.637935 5894 scope.go:122] "RemoveContainer" containerID="a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68" Oct 30 07:54:10.670568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.670544 5894 scope.go:122] "RemoveContainer" containerID="a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68" Oct 30 07:54:10.672946 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:10.672905 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68\": container with ID starting with a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68 not found: ID does not exist" containerID="a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68" Oct 30 07:54:10.673053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.672959 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68"} err="failed to get container status \"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68\": rpc error: code = NotFound desc = could not find container \"a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68\": container with ID starting with a6ac37ea44b969c627516aaf9a82395ecf12f4e0e766d6cb81c1b34126940c68 not found: ID does not exist" Oct 30 07:54:10.692519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692484 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.692739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692600 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.692839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692758 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.692913 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692897 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\" (UniqueName: \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.692980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692902 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:10.692980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.692963 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\" (UniqueName: \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.693087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.693068 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\") pod \"7769b684-807f-49b6-9231-475bc872e040\" (UID: \"7769b684-807f-49b6-9231-475bc872e040\") " Oct 30 07:54:10.693805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.693401 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:10.693805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.693773 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:10.694009 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.693958 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.694242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.694213 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.695595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.695569 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:54:10.695734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.695691 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:10.696487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.696452 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7" pod "7769b684-807f-49b6-9231-475bc872e040" (UID: "7769b684-807f-49b6-9231-475bc872e040"). InnerVolumeSpecName "kube-api-access-844g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:10.795492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.795458 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.795492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.795489 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7769b684-807f-49b6-9231-475bc872e040-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.795492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.795499 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-844g7\" (UniqueName: \"kubernetes.io/projected/7769b684-807f-49b6-9231-475bc872e040-kube-api-access-844g7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.795768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.795508 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7769b684-807f-49b6-9231-475bc872e040-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:10.852452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.852359 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:10.992861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.992827 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:54:10.996706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:10.996666 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fae1d2/test-model-fae1d2-kserve-8ffdf4965-ftlfh"] Oct 30 07:54:12.435801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:12.435769 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7769b684-807f-49b6-9231-475bc872e040" path="/var/lib/kubelet/pods/7769b684-807f-49b6-9231-475bc872e040/volumes" Oct 30 07:54:12.512924 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:12.512881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:14.145368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:14.145329 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:15.835290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:15.835250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:15.877466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:15.877426 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:15.877466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:15.877454 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:17.577851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:17.577811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:19.326764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:19.326723 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:20.695183 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:20.695148 5894 scope.go:122] "RemoveContainer" containerID="e6c886f05fca0bbae76215dda4f397a0631ad9fe5c684bf89f91e2159cf4c2ac" Oct 30 07:54:20.706539 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:20.706512 5894 scope.go:122] "RemoveContainer" containerID="8d80fb0360f85515d997cccccf3eebf8b0830dab00719f3982736e6547171337" Oct 30 07:54:21.172947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:21.172915 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:23.019171 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:23.019135 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:24.783383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.783340 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:54:24.792243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.792022 5894 generic.go:356] "Generic (PLEG): container finished" podID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerID="4165841bfc3fc359b074cd7c32fd8686aa74c1ca5c7cdff249b0afe285caa958" exitCode=0 Oct 30 07:54:24.792243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.792129 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" event={"ID":"feed0dc6-9dcf-4b6f-97d2-54470dce1e39","Type":"ContainerDied","Data":"4165841bfc3fc359b074cd7c32fd8686aa74c1ca5c7cdff249b0afe285caa958"} Oct 30 07:54:24.792243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.792178 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" event={"ID":"feed0dc6-9dcf-4b6f-97d2-54470dce1e39","Type":"ContainerDied","Data":"282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d"} Oct 30 07:54:24.792243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.792203 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="282622eceb4fb744d732934bd04855929a9b6ac5a8f586f82b3978b5f6d6298d" Oct 30 07:54:24.795050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.794892 5894 generic.go:356] "Generic (PLEG): container finished" podID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerID="c8379124a295e72b929eb8ae0738d84308282d7ba6f87d6c655b3c50f8c66b5a" exitCode=0 Oct 30 07:54:24.795050 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.794975 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" event={"ID":"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a","Type":"ContainerDied","Data":"c8379124a295e72b929eb8ae0738d84308282d7ba6f87d6c655b3c50f8c66b5a"} Oct 30 07:54:24.838884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.838854 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:54:24.842750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.842718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:24.930723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.929507 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\") pod \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " Oct 30 07:54:24.930723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.929739 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\" (UniqueName: \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\") pod \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\" (UID: \"feed0dc6-9dcf-4b6f-97d2-54470dce1e39\") " Oct 30 07:54:24.947677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:24.940960 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume" pod "feed0dc6-9dcf-4b6f-97d2-54470dce1e39" (UID: "feed0dc6-9dcf-4b6f-97d2-54470dce1e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:25.044687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.035088 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:25.052254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.045180 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr" pod "feed0dc6-9dcf-4b6f-97d2-54470dce1e39" (UID: "feed0dc6-9dcf-4b6f-97d2-54470dce1e39"). InnerVolumeSpecName "kube-api-access-txwdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:25.055029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.054996 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:25.107086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.104894 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:54:25.136935 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.136900 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwdr\" (UniqueName: \"kubernetes.io/projected/feed0dc6-9dcf-4b6f-97d2-54470dce1e39-kube-api-access-txwdr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:25.231737 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.231700 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws"] Oct 30 07:54:25.233034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233009 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6b7800d6-4988-4410-b320-4b3871eefe57" containerName="revoke-keys" Oct 30 07:54:25.233034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233031 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7800d6-4988-4410-b320-4b3871eefe57" containerName="revoke-keys" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233060 5894 cpu_manager.go:414] "containerMap: removing container" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233065 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233086 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7769b684-807f-49b6-9231-475bc872e040" containerName="main" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233090 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7769b684-807f-49b6-9231-475bc872e040" containerName="main" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233099 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233104 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" Oct 30 07:54:25.233227 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233223 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7800d6-4988-4410-b320-4b3871eefe57" containerName="revoke-keys" Oct 30 07:54:25.233496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233234 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" containerName="payload-processing" Oct 30 07:54:25.233496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233243 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7769b684-807f-49b6-9231-475bc872e040" containerName="main" Oct 30 07:54:25.233496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.233249 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" containerName="payload-pre-processing" Oct 30 07:54:25.236065 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.236044 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.238238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.238205 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\" (UniqueName: \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\") pod \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " Oct 30 07:54:25.238374 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.238330 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\") pod \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\" (UID: \"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a\") " Oct 30 07:54:25.238740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.238706 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume" pod "8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" (UID: "8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:25.239172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.239147 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:25.240754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.240727 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk" pod "8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" (UID: "8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a"). InnerVolumeSpecName "kube-api-access-hn6wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:25.258440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.258361 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:54:25.258440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.258370 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:54:25.258830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.258801 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws"] Oct 30 07:54:25.340610 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.340509 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.340865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.340827 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c955q\" (UniqueName: \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.341007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.340988 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.341167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.341151 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn6wk\" (UniqueName: \"kubernetes.io/projected/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a-kube-api-access-hn6wk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:25.442923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.442885 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.443138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.443116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c955q\" (UniqueName: \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.443260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.443245 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.443670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.443621 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.445693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.445672 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.467420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.467384 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c955q\" (UniqueName: \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\") pod \"maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.565690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.565620 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:25.805696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.805660 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr" Oct 30 07:54:25.806265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.805731 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" event={"ID":"8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a","Type":"ContainerDied","Data":"1eeb1c712d9154a0c5e8bdcd3effb22810ea02a59ef527b3f79c2fce3930a8d2"} Oct 30 07:54:25.806265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.805813 5894 scope.go:122] "RemoveContainer" containerID="c8379124a295e72b929eb8ae0738d84308282d7ba6f87d6c655b3c50f8c66b5a" Oct 30 07:54:25.806265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.806054 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh" Oct 30 07:54:25.904915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.904827 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:54:25.928820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.928771 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-63f24b-7c7ccd6948-ngmpr"] Oct 30 07:54:25.936316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.936286 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws"] Oct 30 07:54:25.964803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.964772 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:54:25.977261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:25.977227 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-63f24b-6b798cd595-b6fmh"] Oct 30 07:54:26.435002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.434961 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a" path="/var/lib/kubelet/pods/8abc5c5a-fe25-40e2-92a9-44d8fb3bd54a/volumes" Oct 30 07:54:26.435482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.435468 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feed0dc6-9dcf-4b6f-97d2-54470dce1e39" path="/var/lib/kubelet/pods/feed0dc6-9dcf-4b6f-97d2-54470dce1e39/volumes" Oct 30 07:54:26.819529 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.819489 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" event={"ID":"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9","Type":"ContainerStarted","Data":"b819871a6c1e9d857ba2487eded82e20a187ed3490ce2c9f83582f33fc36880b"} Oct 30 07:54:26.820002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.819553 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" event={"ID":"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9","Type":"ContainerStarted","Data":"c1dca03f5871f63ce82bbb9d80050535070ffc4e5e862c65dfea79be74bf3af9"} Oct 30 07:54:26.866485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.866421 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" podStartSLOduration=1.866402905 podStartE2EDuration="1.866402905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:54:26.861434611 +0000 UTC m=+3430.552352048" watchObservedRunningTime="2026-10-30 07:54:26.866402905 +0000 UTC m=+3430.557320343" Oct 30 07:54:26.891365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:26.891327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:27.830527 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:27.830488 5894 generic.go:356] "Generic (PLEG): container finished" podID="7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" containerID="b819871a6c1e9d857ba2487eded82e20a187ed3490ce2c9f83582f33fc36880b" exitCode=0 Oct 30 07:54:27.831007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:27.830560 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" event={"ID":"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9","Type":"ContainerDied","Data":"b819871a6c1e9d857ba2487eded82e20a187ed3490ce2c9f83582f33fc36880b"} Oct 30 07:54:28.646922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:28.646882 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:29.015520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.015490 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:29.109859 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.109824 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\") pod \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " Oct 30 07:54:29.110073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.110038 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\") pod \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " Oct 30 07:54:29.110151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.110097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\" (UniqueName: \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\") pod \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\" (UID: \"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9\") " Oct 30 07:54:29.110546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.110516 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca" pod "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" (UID: "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:29.110873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.110850 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:29.112169 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.112143 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca" pod "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" (UID: "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:54:29.112482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.112457 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q" pod "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" (UID: "7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9"). InnerVolumeSpecName "kube-api-access-c955q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:29.212535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.212455 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c955q\" (UniqueName: \"kubernetes.io/projected/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-kube-api-access-c955q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:29.212535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.212486 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:29.851181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.851147 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" Oct 30 07:54:29.851181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.851168 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws" event={"ID":"7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9","Type":"ContainerDied","Data":"c1dca03f5871f63ce82bbb9d80050535070ffc4e5e862c65dfea79be74bf3af9"} Oct 30 07:54:29.851412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:29.851217 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1dca03f5871f63ce82bbb9d80050535070ffc4e5e862c65dfea79be74bf3af9" Oct 30 07:54:30.366936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:30.366897 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:32.219393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:32.219353 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:32.335649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:32.335599 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:33.946275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:33.946225 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:34.833003 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:34.832962 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:54:35.509879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:35.509844 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:54:35.546000 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:35.545965 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:54:35.559230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:35.559193 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:35.727008 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:35.726976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:35.850975 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:35.850922 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:37.434256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:37.434217 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:39.220861 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:39.220815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:39.849170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:39.849118 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" podUID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" containerName="main" containerID="cri-o://e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d" gracePeriod=45 Oct 30 07:54:40.170071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.170039 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:54:40.340232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340194 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.340232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340238 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340271 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340434 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\" (UniqueName: \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340540 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340564 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340584 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:40.340780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340694 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\") pod \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\" (UID: \"3cc35c21-a325-46d3-8fba-7fe5516a2fb1\") " Oct 30 07:54:40.341107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.340934 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:40.341445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.341419 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.341531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.341451 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.341531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.341467 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.342661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.342609 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:54:40.342762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.342739 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "kube-api-access-n255b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:40.343037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.343022 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm" pod "3cc35c21-a325-46d3-8fba-7fe5516a2fb1" (UID: "3cc35c21-a325-46d3-8fba-7fe5516a2fb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:40.442593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.442566 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n255b\" (UniqueName: \"kubernetes.io/projected/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-kube-api-access-n255b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.442715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.442600 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.442715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.442619 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cc35c21-a325-46d3-8fba-7fe5516a2fb1-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.510955 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.510914 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" podUID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" containerName="payload-processing" containerID="cri-o://abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13" gracePeriod=25 Oct 30 07:54:40.546488 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.546430 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" containerID="cri-o://439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3" gracePeriod=25 Oct 30 07:54:40.782441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.782404 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:54:40.861965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.861924 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\" (UniqueName: \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\") pod \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " Oct 30 07:54:40.862167 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.862100 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\") pod \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\" (UID: \"dd0c4c56-76ef-4a02-a27f-52033755b4a2\") " Oct 30 07:54:40.862627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.862567 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume" pod "dd0c4c56-76ef-4a02-a27f-52033755b4a2" (UID: "dd0c4c56-76ef-4a02-a27f-52033755b4a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:40.862944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.862920 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0c4c56-76ef-4a02-a27f-52033755b4a2-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.867162 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.867130 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t" pod "dd0c4c56-76ef-4a02-a27f-52033755b4a2" (UID: "dd0c4c56-76ef-4a02-a27f-52033755b4a2"). InnerVolumeSpecName "kube-api-access-6kn7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:40.921600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.921563 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:40.954409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.952360 5894 generic.go:356] "Generic (PLEG): container finished" podID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" containerID="abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13" exitCode=0 Oct 30 07:54:40.954409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.952605 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" event={"ID":"dd0c4c56-76ef-4a02-a27f-52033755b4a2","Type":"ContainerDied","Data":"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13"} Oct 30 07:54:40.954409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.952784 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" event={"ID":"dd0c4c56-76ef-4a02-a27f-52033755b4a2","Type":"ContainerDied","Data":"f8102ef55be81723304702ced50bfdc27a4e5246c3d9eb63e84daf2255b1911b"} Oct 30 07:54:40.954409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.952826 5894 scope.go:122] "RemoveContainer" containerID="abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13" Oct 30 07:54:40.954409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.953031 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft" Oct 30 07:54:40.970793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.970622 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kn7t\" (UniqueName: \"kubernetes.io/projected/dd0c4c56-76ef-4a02-a27f-52033755b4a2-kube-api-access-6kn7t\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:40.988878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.988845 5894 generic.go:356] "Generic (PLEG): container finished" podID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" containerID="e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d" exitCode=0 Oct 30 07:54:40.989058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.988951 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" event={"ID":"3cc35c21-a325-46d3-8fba-7fe5516a2fb1","Type":"ContainerDied","Data":"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d"} Oct 30 07:54:40.989058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.988978 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" event={"ID":"3cc35c21-a325-46d3-8fba-7fe5516a2fb1","Type":"ContainerDied","Data":"dfccac8bf00eacbe75a4f1ecf98be1f391d0fc465ef005073776f4b91c8eb4b9"} Oct 30 07:54:40.989058 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:40.989038 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf" Oct 30 07:54:41.010334 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.010308 5894 scope.go:122] "RemoveContainer" containerID="abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13" Oct 30 07:54:41.010787 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:41.010758 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13\": container with ID starting with abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13 not found: ID does not exist" containerID="abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13" Oct 30 07:54:41.010911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.010794 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13"} err="failed to get container status \"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13\": rpc error: code = NotFound desc = could not find container \"abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13\": container with ID starting with abfa56ec4c9716774edcc024f9b96aa4370795e32e64871f25bda3c6ab4b5e13 not found: ID does not exist" Oct 30 07:54:41.010911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.010820 5894 scope.go:122] "RemoveContainer" containerID="e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d" Oct 30 07:54:41.027045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.027016 5894 scope.go:122] "RemoveContainer" containerID="e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d" Oct 30 07:54:41.027398 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:41.027373 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d\": container with ID starting with e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d not found: ID does not exist" containerID="e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d" Oct 30 07:54:41.027467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.027410 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d"} err="failed to get container status \"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d\": rpc error: code = NotFound desc = could not find container \"e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d\": container with ID starting with e4f33f7e58d69f6854575a6611122cf74b0deb654d0d29560647db0751c1f83d not found: ID does not exist" Oct 30 07:54:41.036089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.036053 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:54:41.040080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.040046 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fae1d2-855b6f4bc4-7xvft"] Oct 30 07:54:41.075748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.075712 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:54:41.081974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:41.081945 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-61f6c2/test-model-61f6c2-kserve-c66b859ff-pvkkf"] Oct 30 07:54:42.434748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:42.434713 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" path="/var/lib/kubelet/pods/3cc35c21-a325-46d3-8fba-7fe5516a2fb1/volumes" Oct 30 07:54:42.435242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:42.435220 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" path="/var/lib/kubelet/pods/dd0c4c56-76ef-4a02-a27f-52033755b4a2/volumes" Oct 30 07:54:42.633952 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:42.633917 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:42.883024 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:42.882981 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:43.234982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:43.234894 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:44.328312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:44.328272 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:45.450545 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:45.450508 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:54:45.452258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:45.450845 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" containerID="cri-o://5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43" gracePeriod=30 Oct 30 07:54:46.078618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:46.078574 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:48.065661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:48.065613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:49.254698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:49.254659 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:54:49.255159 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:49.254861 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" containerID="cri-o://79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23" gracePeriod=30 Oct 30 07:54:49.269015 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:49.268976 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:54:49.269218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:49.269192 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" containerID="cri-o://f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a" gracePeriod=30 Oct 30 07:54:49.609677 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:49.609615 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:50.209593 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.209551 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:50.276808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.276762 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:54:50.277220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.276843 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:54:50.331902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.331861 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:50.361011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.360974 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5"] Oct 30 07:54:50.370512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.370483 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-63f24b-qljj5"] Oct 30 07:54:50.435168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.435136 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adefa29-ed2c-4bc7-b9e7-b4643a31904b" path="/var/lib/kubelet/pods/3adefa29-ed2c-4bc7-b9e7-b4643a31904b/volumes" Oct 30 07:54:50.484806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.484718 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:50.756513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.756411 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:54:50.842222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.842189 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:50.872984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.872950 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873027 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873136 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873184 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873174 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873193 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873221 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873216 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.873368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873299 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.873368 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873344 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873467 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873473 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.873685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873504 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\") pod \"7b8e79fc-f42e-4162-9826-429e9181c65c\" (UID: \"7b8e79fc-f42e-4162-9826-429e9181c65c\") " Oct 30 07:54:50.873685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873619 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:50.873995 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.873969 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.874339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.874320 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.874451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.874345 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.874451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.874359 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.874451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.874375 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.874451 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.874390 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8e79fc-f42e-4162-9826-429e9181c65c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.875722 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.875694 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:50.875836 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.875818 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:50.876074 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.876058 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:50.877289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.877264 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28" pod "7b8e79fc-f42e-4162-9826-429e9181c65c" (UID: "7b8e79fc-f42e-4162-9826-429e9181c65c"). InnerVolumeSpecName "kube-api-access-cvt28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:50.975997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.975950 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.975997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.975995 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvt28\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-kube-api-access-cvt28\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.976233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.976007 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:50.976233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:50.976020 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8e79fc-f42e-4162-9826-429e9181c65c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:51.097476 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.097438 5894 generic.go:356] "Generic (PLEG): container finished" podID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerID="5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43" exitCode=0 Oct 30 07:54:51.097705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.097497 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" Oct 30 07:54:51.097705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.097486 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" event={"ID":"7b8e79fc-f42e-4162-9826-429e9181c65c","Type":"ContainerDied","Data":"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43"} Oct 30 07:54:51.097705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.097653 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf" event={"ID":"7b8e79fc-f42e-4162-9826-429e9181c65c","Type":"ContainerDied","Data":"b7d5aeafbc88741ef86b5c3471ed5dddbac044ad7912ffcf1afd43dafbfe8fc7"} Oct 30 07:54:51.097705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.097685 5894 scope.go:122] "RemoveContainer" containerID="5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43" Oct 30 07:54:51.112007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.111987 5894 scope.go:122] "RemoveContainer" containerID="5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43" Oct 30 07:54:51.112309 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:51.112284 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43\": container with ID starting with 5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43 not found: ID does not exist" containerID="5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43" Oct 30 07:54:51.112424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.112316 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43"} err="failed to get container status \"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43\": rpc error: code = NotFound desc = could not find container \"5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43\": container with ID starting with 5f741c9d8462c573a40bed532ec88228077d8f35500a0a1c3a542345ecc80e43 not found: ID does not exist" Oct 30 07:54:51.153243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.153207 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:54:51.165309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:51.165276 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-ng5gf"] Oct 30 07:54:52.145079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:52.145033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:52.440685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:52.440566 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" path="/var/lib/kubelet/pods/7b8e79fc-f42e-4162-9826-429e9181c65c/volumes" Oct 30 07:54:53.235422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:53.235374 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:54:53.939795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:53.939730 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:54.582645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.582607 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:54:54.603776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.603749 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:54:54.666535 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666490 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.666803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666600 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.666803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666677 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.666803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666709 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.666803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666781 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666821 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666889 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666929 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.666969 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667077 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667133 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667171 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667208 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667301 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667276 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667361 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667410 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\") pod \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\" (UID: \"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9\") " Oct 30 07:54:54.667606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667463 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.667606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.667494 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\") pod \"fd360e22-b736-427d-a64e-3e8c1c691cc7\" (UID: \"fd360e22-b736-427d-a64e-3e8c1c691cc7\") " Oct 30 07:54:54.668144 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668090 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668169 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668178 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668253 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668361 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668401 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.668549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:54.668735 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.668710 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.669872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669844 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669875 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669892 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669906 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669917 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669929 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd360e22-b736-427d-a64e-3e8c1c691cc7-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669942 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.669982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.669955 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.670495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.670260 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.671089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.671063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:54:54.671261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.671235 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.671348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.671325 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:54.673087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.672990 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:54.673087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.673046 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "kube-api-access-wm4p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:54.673264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.673091 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "kube-api-access-m6fws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:54.673466 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.673442 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo" pod "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" (UID: "dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:54:54.674324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.674301 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:54:54.674542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.674511 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy" pod "fd360e22-b736-427d-a64e-3e8c1c691cc7" (UID: "fd360e22-b736-427d-a64e-3e8c1c691cc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:54:54.771514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771482 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771512 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771521 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6fws\" (UniqueName: \"kubernetes.io/projected/fd360e22-b736-427d-a64e-3e8c1c691cc7-kube-api-access-m6fws\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771532 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm4p4\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-kube-api-access-wm4p4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771543 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771551 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd360e22-b736-427d-a64e-3e8c1c691cc7-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771560 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771567 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771574 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:54.771809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:54.771581 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:54:55.133417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.133380 5894 generic.go:356] "Generic (PLEG): container finished" podID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerID="f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a" exitCode=0 Oct 30 07:54:55.133605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.133429 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" Oct 30 07:54:55.133605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.133457 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" event={"ID":"fd360e22-b736-427d-a64e-3e8c1c691cc7","Type":"ContainerDied","Data":"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a"} Oct 30 07:54:55.133605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.133512 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh" event={"ID":"fd360e22-b736-427d-a64e-3e8c1c691cc7","Type":"ContainerDied","Data":"4dcb0dd370a2f4ab3b5f7c3f863444f702ba53e5eba432e6936575c16ba74208"} Oct 30 07:54:55.133605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.133539 5894 scope.go:122] "RemoveContainer" containerID="f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a" Oct 30 07:54:55.135829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.135678 5894 generic.go:356] "Generic (PLEG): container finished" podID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerID="79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23" exitCode=0 Oct 30 07:54:55.135829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.135731 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" event={"ID":"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9","Type":"ContainerDied","Data":"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23"} Oct 30 07:54:55.135829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.135746 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" Oct 30 07:54:55.135829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.135794 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz" event={"ID":"dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9","Type":"ContainerDied","Data":"5791efabe78eb615a89c312f24e5ce2dc3fc1e84a548ef4a50b4e255caadb6cb"} Oct 30 07:54:55.146378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.146354 5894 scope.go:122] "RemoveContainer" containerID="f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a" Oct 30 07:54:55.146706 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:55.146679 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a\": container with ID starting with f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a not found: ID does not exist" containerID="f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a" Oct 30 07:54:55.146822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.146713 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a"} err="failed to get container status \"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a\": rpc error: code = NotFound desc = could not find container \"f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a\": container with ID starting with f36a81147906de878ff447ee8b7fdf5c27f78fff4bea9571b1c4e25e926f9a0a not found: ID does not exist" Oct 30 07:54:55.146822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.146730 5894 scope.go:122] "RemoveContainer" containerID="79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23" Oct 30 07:54:55.159899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.159873 5894 scope.go:122] "RemoveContainer" containerID="79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23" Oct 30 07:54:55.160210 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:54:55.160191 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23\": container with ID starting with 79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23 not found: ID does not exist" containerID="79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23" Oct 30 07:54:55.160284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.160220 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23"} err="failed to get container status \"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23\": rpc error: code = NotFound desc = could not find container \"79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23\": container with ID starting with 79e226cb6ff4318ed53cc4b15e4447414c18b86bde167831b07ade2ea5c3cf23 not found: ID does not exist" Oct 30 07:54:55.190566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.190531 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:54:55.203119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.203081 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-sgrmz"] Oct 30 07:54:55.228006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.227973 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:54:55.231664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.231627 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-vlkzh"] Oct 30 07:54:55.628042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:55.628010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:56.434013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:56.433969 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" path="/var/lib/kubelet/pods/dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9/volumes" Oct 30 07:54:56.434611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:56.434586 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" path="/var/lib/kubelet/pods/fd360e22-b736-427d-a64e-3e8c1c691cc7/volumes" Oct 30 07:54:56.900136 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:56.900103 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:57.019578 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:57.019547 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:57.298090 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:57.298056 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:58.184441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:58.184406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:58.336086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:58.336051 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:58.462506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:58.462423 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:54:59.020726 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:54:59.020685 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:00.720202 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.720168 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:00.874999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.874971 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:55:00.986017 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.985980 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\") pod \"8e2d78cd-f433-4591-ad91-28935165fe2f\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " Oct 30 07:55:00.986224 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.986115 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\" (UniqueName: \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\") pod \"8e2d78cd-f433-4591-ad91-28935165fe2f\" (UID: \"8e2d78cd-f433-4591-ad91-28935165fe2f\") " Oct 30 07:55:00.986398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.986369 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume" pod "8e2d78cd-f433-4591-ad91-28935165fe2f" (UID: "8e2d78cd-f433-4591-ad91-28935165fe2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:00.986568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.986554 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e2d78cd-f433-4591-ad91-28935165fe2f-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:00.988353 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:00.988324 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt" pod "8e2d78cd-f433-4591-ad91-28935165fe2f" (UID: "8e2d78cd-f433-4591-ad91-28935165fe2f"). InnerVolumeSpecName "kube-api-access-4tpgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:01.087895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.087803 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpgt\" (UniqueName: \"kubernetes.io/projected/8e2d78cd-f433-4591-ad91-28935165fe2f-kube-api-access-4tpgt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:01.195823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.195785 5894 generic.go:356] "Generic (PLEG): container finished" podID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerID="439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3" exitCode=0 Oct 30 07:55:01.195994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.195829 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" Oct 30 07:55:01.195994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.195882 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" event={"ID":"8e2d78cd-f433-4591-ad91-28935165fe2f","Type":"ContainerDied","Data":"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3"} Oct 30 07:55:01.195994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.195975 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw" event={"ID":"8e2d78cd-f433-4591-ad91-28935165fe2f","Type":"ContainerDied","Data":"320c4bc88f5a15477ee16114f96a5506786a8515d2f70ec01dd1e67178aef042"} Oct 30 07:55:01.196186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.196007 5894 scope.go:122] "RemoveContainer" containerID="439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3" Oct 30 07:55:01.208497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.208479 5894 scope.go:122] "RemoveContainer" containerID="439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3" Oct 30 07:55:01.208797 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:55:01.208774 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3\": container with ID starting with 439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3 not found: ID does not exist" containerID="439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3" Oct 30 07:55:01.208867 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.208809 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3"} err="failed to get container status \"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3\": rpc error: code = NotFound desc = could not find container \"439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3\": container with ID starting with 439b0a8bcc3c028145d0ce921bc8b4625996a7a50c05562f6184e1e244a91ff3 not found: ID does not exist" Oct 30 07:55:01.250265 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.250231 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:55:01.253809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:01.253782 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fae1d2-5949b84c55-xb7fw"] Oct 30 07:55:02.433689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:02.433649 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" path="/var/lib/kubelet/pods/8e2d78cd-f433-4591-ad91-28935165fe2f/volumes" Oct 30 07:55:02.440100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:02.440074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:04.187431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:04.187391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:04.832860 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:55:04.832819 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:55:05.885891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:05.885843 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:07.627270 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:07.627232 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:09.229012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:09.228977 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:10.824344 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:10.824304 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:12.651715 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:12.651628 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:14.131842 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:14.131808 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:15.811803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:15.811758 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:17.223010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.222975 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz"] Oct 30 07:55:17.223806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223788 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" Oct 30 07:55:17.223806 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223805 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223821 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223826 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223846 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" containerName="main" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223851 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" containerName="main" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223870 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" containerName="revoke-keys" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223875 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" containerName="revoke-keys" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223884 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223888 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223904 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" containerName="payload-processing" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223909 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" containerName="payload-processing" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223920 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" Oct 30 07:55:17.223939 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.223925 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224027 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc4286cf-ea0a-4c54-a5e5-632c81fe2ed9" containerName="istio-proxy" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224041 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2d78cd-f433-4591-ad91-28935165fe2f" containerName="payload-pre-processing" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224048 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd360e22-b736-427d-a64e-3e8c1c691cc7" containerName="istio-proxy" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224057 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cc35c21-a325-46d3-8fba-7fe5516a2fb1" containerName="main" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224065 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" containerName="revoke-keys" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224070 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8e79fc-f42e-4162-9826-429e9181c65c" containerName="istio-proxy" Oct 30 07:55:17.224323 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.224079 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0c4c56-76ef-4a02-a27f-52033755b4a2" containerName="payload-processing" Oct 30 07:55:17.229606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.229577 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.252338 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.252303 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz"] Oct 30 07:55:17.260158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.260129 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:55:17.260339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.260177 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:55:17.334725 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.334688 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.334896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.334750 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkb24\" (UniqueName: \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.334958 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.334919 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.436936 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.436894 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.437152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.436960 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkb24\" (UniqueName: \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.437152 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.437079 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.437669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.437597 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.440239 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.440215 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.462087 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.462055 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkb24\" (UniqueName: \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\") pod \"maas-api-revoke-sub-41aef6d6-pkqmz\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.500597 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.500507 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:17.564718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.564685 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:17.883551 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:55:17.883520 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424488d6_cb5f_472d_b53d_978f5cc918fa.slice/crio-a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271 WatchSource:0}: Error finding container a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271: Status 404 returned error can't find the container with id a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271 Oct 30 07:55:17.884881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:17.884852 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz"] Oct 30 07:55:18.382807 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:18.382765 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" event={"ID":"424488d6-cb5f-472d-b53d-978f5cc918fa","Type":"ContainerStarted","Data":"a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271"} Oct 30 07:55:18.969389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:18.969345 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:19.423658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.421178 5894 generic.go:356] "Generic (PLEG): container finished" podID="424488d6-cb5f-472d-b53d-978f5cc918fa" containerID="861e988edb17746371a8498b6ccc75373737b35bd5576a82e77166a6a55add6f" exitCode=0 Oct 30 07:55:19.423658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.421310 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" event={"ID":"424488d6-cb5f-472d-b53d-978f5cc918fa","Type":"ContainerDied","Data":"861e988edb17746371a8498b6ccc75373737b35bd5576a82e77166a6a55add6f"} Oct 30 07:55:19.509445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.509409 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:19.624490 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.624453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:19.641676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.641620 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws"] Oct 30 07:55:19.645154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.645119 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fae1d2-sf7ws"] Oct 30 07:55:19.754309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:19.754221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:20.164706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.164673 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:20.438175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.438137 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9" path="/var/lib/kubelet/pods/7aa3cfdf-55c1-4188-a9cc-b83bc41c27f9/volumes" Oct 30 07:55:20.603321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.603292 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:20.626885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.626853 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\") pod \"424488d6-cb5f-472d-b53d-978f5cc918fa\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " Oct 30 07:55:20.627091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.626923 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\" (UniqueName: \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\") pod \"424488d6-cb5f-472d-b53d-978f5cc918fa\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " Oct 30 07:55:20.627091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.626977 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\") pod \"424488d6-cb5f-472d-b53d-978f5cc918fa\" (UID: \"424488d6-cb5f-472d-b53d-978f5cc918fa\") " Oct 30 07:55:20.627420 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.627392 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca" pod "424488d6-cb5f-472d-b53d-978f5cc918fa" (UID: "424488d6-cb5f-472d-b53d-978f5cc918fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:20.627736 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.627705 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:20.629148 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.629123 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca" pod "424488d6-cb5f-472d-b53d-978f5cc918fa" (UID: "424488d6-cb5f-472d-b53d-978f5cc918fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:20.629248 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.629161 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24" pod "424488d6-cb5f-472d-b53d-978f5cc918fa" (UID: "424488d6-cb5f-472d-b53d-978f5cc918fa"). InnerVolumeSpecName "kube-api-access-xkb24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:20.729118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.729025 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/424488d6-cb5f-472d-b53d-978f5cc918fa-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:20.729118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.729065 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkb24\" (UniqueName: \"kubernetes.io/projected/424488d6-cb5f-472d-b53d-978f5cc918fa-kube-api-access-xkb24\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:20.759482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.759449 5894 scope.go:122] "RemoveContainer" containerID="25b0e63a336b9af847e75ae6f141f9d515cff8ee7a3233496a1a156f584c86e5" Oct 30 07:55:20.771340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.771311 5894 scope.go:122] "RemoveContainer" containerID="56f239b7f6683b5be44ca919bab3e4061cd3e75b4ee79ab57a4d79089f5596be" Oct 30 07:55:20.782119 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.782090 5894 scope.go:122] "RemoveContainer" containerID="4165841bfc3fc359b074cd7c32fd8686aa74c1ca5c7cdff249b0afe285caa958" Oct 30 07:55:20.796185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.796159 5894 scope.go:122] "RemoveContainer" containerID="e1ecd98229046d5714504ec34ab721e5a1df9cf3a8dd958d803a866b6f65f56d" Oct 30 07:55:20.809082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.809055 5894 scope.go:122] "RemoveContainer" containerID="fb80a813ed1a718a510fd456aed24743ab870aaa172511076b2b8998b7452470" Oct 30 07:55:20.820067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.820043 5894 scope.go:122] "RemoveContainer" containerID="a420df179d4aa58d5263c56411b94d0aa0130d41693c258ce2b60f01bcd51718" Oct 30 07:55:20.831053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.831031 5894 scope.go:122] "RemoveContainer" containerID="deff1a7ea6406575624473abb64da3e73f0063a9e742daaf2ebbd729872f9c79" Oct 30 07:55:20.842477 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:20.842452 5894 scope.go:122] "RemoveContainer" containerID="29aecdf0cdeb20a5babedbeb68af81deb177f3170f5839a2ddd0b3d94c0a426d" Oct 30 07:55:21.440185 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:21.440156 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" Oct 30 07:55:21.440721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:21.440145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz" event={"ID":"424488d6-cb5f-472d-b53d-978f5cc918fa","Type":"ContainerDied","Data":"a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271"} Oct 30 07:55:21.440721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:21.440276 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8833908e533b518f4e009f2760b309bfea62c55c9511dad010320a26f7bc271" Oct 30 07:55:21.505774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:21.505735 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:23.105740 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:23.105701 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:24.749349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:24.749311 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:26.414778 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:26.414744 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:26.568793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:26.568755 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:26.777034 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:26.776932 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:27.039724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.038088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:27.617442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.617406 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9"] Oct 30 07:55:27.618231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.618214 5894 cpu_manager.go:414] "containerMap: removing container" podUID="424488d6-cb5f-472d-b53d-978f5cc918fa" containerName="revoke-keys" Oct 30 07:55:27.618231 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.618231 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="424488d6-cb5f-472d-b53d-978f5cc918fa" containerName="revoke-keys" Oct 30 07:55:27.618408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.618398 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="424488d6-cb5f-472d-b53d-978f5cc918fa" containerName="revoke-keys" Oct 30 07:55:27.624438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.624414 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.643771 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.643740 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9"] Oct 30 07:55:27.680896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.680852 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:55:27.681098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.680945 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:55:27.764619 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.764585 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.764811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.764736 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.764811 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.764767 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b66t\" (UniqueName: \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.866118 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.866082 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.866309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.866164 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.866309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.866184 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b66t\" (UniqueName: \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.866777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.866754 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.868728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.868676 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.890408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.890367 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b66t\" (UniqueName: \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\") pod \"maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:27.939881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:27.939845 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:28.255663 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.254746 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:55:28.264690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.263276 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9"] Oct 30 07:55:28.298104 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.298071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:28.446746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.446712 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:28.523721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.523558 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" event={"ID":"de886e71-0319-4c22-9df8-036f43e8072c","Type":"ContainerStarted","Data":"8078a2865c85e910b0fd6bef1e8a63c61be019278eda5e1d0932ceb1802cd612"} Oct 30 07:55:28.898002 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.897955 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9"] Oct 30 07:55:28.903340 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.903309 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:28.924076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:28.924040 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9"] Oct 30 07:55:29.017848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.017810 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.018041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.018021 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxfsb\" (UniqueName: \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.018191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.018176 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.119392 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.119354 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.119589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.119527 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxfsb\" (UniqueName: \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.119738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.119719 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.120432 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.120406 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.122020 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.121992 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.144053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.144021 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxfsb\" (UniqueName: \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\") pod \"maas-api-revoke-sub-a3366f68-x4wr9\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.224455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.224361 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:29.536059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.535017 5894 generic.go:356] "Generic (PLEG): container finished" podID="de886e71-0319-4c22-9df8-036f43e8072c" containerID="2bebb0fbeb760ec89c9d885747b09faf52b91feef79ffbecd8f9a14cae9aa9e9" exitCode=0 Oct 30 07:55:29.536059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.535067 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" event={"ID":"de886e71-0319-4c22-9df8-036f43e8072c","Type":"ContainerDied","Data":"2bebb0fbeb760ec89c9d885747b09faf52b91feef79ffbecd8f9a14cae9aa9e9"} Oct 30 07:55:29.540333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:29.540309 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9"] Oct 30 07:55:29.541158 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:55:29.541133 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73d760a_cdbb_4b6a_a894_ee593c427522.slice/crio-0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf WatchSource:0}: Error finding container 0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf: Status 404 returned error can't find the container with id 0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf Oct 30 07:55:30.230694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.230552 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:30.545021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.544988 5894 generic.go:356] "Generic (PLEG): container finished" podID="b73d760a-cdbb-4b6a-a894-ee593c427522" containerID="bc768bb438423670da121a99dcf2b81a341f4fa0fb429141eef3c4f1b5411f8e" exitCode=0 Oct 30 07:55:30.545200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.545032 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" event={"ID":"b73d760a-cdbb-4b6a-a894-ee593c427522","Type":"ContainerDied","Data":"bc768bb438423670da121a99dcf2b81a341f4fa0fb429141eef3c4f1b5411f8e"} Oct 30 07:55:30.545200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.545090 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" event={"ID":"b73d760a-cdbb-4b6a-a894-ee593c427522","Type":"ContainerStarted","Data":"0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf"} Oct 30 07:55:30.718794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.718764 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:30.740012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.739982 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\") pod \"de886e71-0319-4c22-9df8-036f43e8072c\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " Oct 30 07:55:30.740174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.740029 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\") pod \"de886e71-0319-4c22-9df8-036f43e8072c\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " Oct 30 07:55:30.740174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.740127 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\" (UniqueName: \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\") pod \"de886e71-0319-4c22-9df8-036f43e8072c\" (UID: \"de886e71-0319-4c22-9df8-036f43e8072c\") " Oct 30 07:55:30.740379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.740348 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca" pod "de886e71-0319-4c22-9df8-036f43e8072c" (UID: "de886e71-0319-4c22-9df8-036f43e8072c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:30.740820 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.740798 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:30.742359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.742334 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t" pod "de886e71-0319-4c22-9df8-036f43e8072c" (UID: "de886e71-0319-4c22-9df8-036f43e8072c"). InnerVolumeSpecName "kube-api-access-6b66t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:30.742359 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.742348 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca" pod "de886e71-0319-4c22-9df8-036f43e8072c" (UID: "de886e71-0319-4c22-9df8-036f43e8072c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:30.842029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.841991 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de886e71-0319-4c22-9df8-036f43e8072c-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:30.842029 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:30.842029 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b66t\" (UniqueName: \"kubernetes.io/projected/de886e71-0319-4c22-9df8-036f43e8072c-kube-api-access-6b66t\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:31.554841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.554811 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" Oct 30 07:55:31.555264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.554844 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9" event={"ID":"de886e71-0319-4c22-9df8-036f43e8072c","Type":"ContainerDied","Data":"8078a2865c85e910b0fd6bef1e8a63c61be019278eda5e1d0932ceb1802cd612"} Oct 30 07:55:31.555264 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.554894 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8078a2865c85e910b0fd6bef1e8a63c61be019278eda5e1d0932ceb1802cd612" Oct 30 07:55:31.733339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.733311 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:31.751679 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.751619 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\") pod \"b73d760a-cdbb-4b6a-a894-ee593c427522\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " Oct 30 07:55:31.751856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.751779 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\" (UniqueName: \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\") pod \"b73d760a-cdbb-4b6a-a894-ee593c427522\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " Oct 30 07:55:31.751932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.751909 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\") pod \"b73d760a-cdbb-4b6a-a894-ee593c427522\" (UID: \"b73d760a-cdbb-4b6a-a894-ee593c427522\") " Oct 30 07:55:31.752007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.751981 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca" pod "b73d760a-cdbb-4b6a-a894-ee593c427522" (UID: "b73d760a-cdbb-4b6a-a894-ee593c427522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:31.752383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.752366 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:31.753944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.753917 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca" pod "b73d760a-cdbb-4b6a-a894-ee593c427522" (UID: "b73d760a-cdbb-4b6a-a894-ee593c427522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:31.754051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.753980 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb" pod "b73d760a-cdbb-4b6a-a894-ee593c427522" (UID: "b73d760a-cdbb-4b6a-a894-ee593c427522"). InnerVolumeSpecName "kube-api-access-xxfsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:31.853688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.853594 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxfsb\" (UniqueName: \"kubernetes.io/projected/b73d760a-cdbb-4b6a-a894-ee593c427522-kube-api-access-xxfsb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:31.853688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.853649 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73d760a-cdbb-4b6a-a894-ee593c427522-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:31.921902 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:31.921865 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:32.392803 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:32.392762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:32.564797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:32.564760 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" Oct 30 07:55:32.565284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:32.564793 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9" event={"ID":"b73d760a-cdbb-4b6a-a894-ee593c427522","Type":"ContainerDied","Data":"0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf"} Oct 30 07:55:32.565284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:32.564845 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5d5f7b555a010ace801fe4ee7dcba3cc50bf68b5abcadffcfa208d8e3b56cf" Oct 30 07:55:33.715267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:33.715233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:34.832897 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:55:34.832852 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:55:35.048375 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:35.048337 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:55:35.390669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:35.390597 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:37.252330 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:37.252290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:38.133308 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:38.133265 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:55:38.265901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:38.265680 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:38.398028 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:38.397747 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:39.173519 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:39.173486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:41.023808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:41.023770 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:42.671358 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:42.671314 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:43.133894 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.133827 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" containerID="cri-o://51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab" gracePeriod=25 Oct 30 07:55:43.293404 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.293370 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" podUID="860fba1b-d47b-46bb-85a5-48b1536660a6" containerName="main" containerID="cri-o://1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d" gracePeriod=45 Oct 30 07:55:43.593195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.593163 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:55:43.660772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.660725 5894 generic.go:356] "Generic (PLEG): container finished" podID="860fba1b-d47b-46bb-85a5-48b1536660a6" containerID="1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d" exitCode=0 Oct 30 07:55:43.660993 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.660858 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" Oct 30 07:55:43.661068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.661021 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" event={"ID":"860fba1b-d47b-46bb-85a5-48b1536660a6","Type":"ContainerDied","Data":"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d"} Oct 30 07:55:43.661127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.661069 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5" event={"ID":"860fba1b-d47b-46bb-85a5-48b1536660a6","Type":"ContainerDied","Data":"116b8c1c33cdd9f548d40b35ae582df89ef4f8cf0c0a93991b770708039fa046"} Oct 30 07:55:43.661127 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.661093 5894 scope.go:122] "RemoveContainer" containerID="1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d" Oct 30 07:55:43.673187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.673162 5894 scope.go:122] "RemoveContainer" containerID="1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d" Oct 30 07:55:43.673698 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:55:43.673518 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d\": container with ID starting with 1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d not found: ID does not exist" containerID="1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d" Oct 30 07:55:43.673698 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.673548 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d"} err="failed to get container status \"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d\": rpc error: code = NotFound desc = could not find container \"1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d\": container with ID starting with 1b80721cee56d69d9a849ffe315b7d5931fb3f2737b06fac325f31d7423f350d not found: ID does not exist" Oct 30 07:55:43.684079 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684048 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684122 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\" (UniqueName: \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684343 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684407 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684484 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684468 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\") pod \"860fba1b-d47b-46bb-85a5-48b1536660a6\" (UID: \"860fba1b-d47b-46bb-85a5-48b1536660a6\") " Oct 30 07:55:43.684888 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684345 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:43.685088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.684507 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:43.685088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.685042 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:43.685393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.685343 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:43.685393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.685367 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:43.685393 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.685383 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:43.689509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.688008 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:55:43.689509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.688111 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "kube-api-access-7szcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:43.689905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.689877 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs" pod "860fba1b-d47b-46bb-85a5-48b1536660a6" (UID: "860fba1b-d47b-46bb-85a5-48b1536660a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:43.786575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.786535 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/860fba1b-d47b-46bb-85a5-48b1536660a6-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:43.786575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.786571 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/860fba1b-d47b-46bb-85a5-48b1536660a6-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:43.786575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:43.786582 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szcd\" (UniqueName: \"kubernetes.io/projected/860fba1b-d47b-46bb-85a5-48b1536660a6-kube-api-access-7szcd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:44.015003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:44.014960 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:55:44.018828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:44.018794 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-fc3a23/test-model-fc3a23-kserve-8655746c79-2h4z5"] Oct 30 07:55:44.344880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:44.344844 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:44.435051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:44.435016 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860fba1b-d47b-46bb-85a5-48b1536660a6" path="/var/lib/kubelet/pods/860fba1b-d47b-46bb-85a5-48b1536660a6/volumes" Oct 30 07:55:45.971312 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:45.971277 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:47.693980 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:47.693944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:49.422869 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:49.422828 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:50.677549 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:50.677513 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:55:50.733263 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:50.733227 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:51.121987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.121952 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm"] Oct 30 07:55:51.123010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.122982 5894 cpu_manager.go:414] "containerMap: removing container" podUID="de886e71-0319-4c22-9df8-036f43e8072c" containerName="revoke-keys" Oct 30 07:55:51.123010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123012 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de886e71-0319-4c22-9df8-036f43e8072c" containerName="revoke-keys" Oct 30 07:55:51.123223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123050 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b73d760a-cdbb-4b6a-a894-ee593c427522" containerName="revoke-keys" Oct 30 07:55:51.123223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123058 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73d760a-cdbb-4b6a-a894-ee593c427522" containerName="revoke-keys" Oct 30 07:55:51.123223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123100 5894 cpu_manager.go:414] "containerMap: removing container" podUID="860fba1b-d47b-46bb-85a5-48b1536660a6" containerName="main" Oct 30 07:55:51.123223 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123107 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860fba1b-d47b-46bb-85a5-48b1536660a6" containerName="main" Oct 30 07:55:51.123387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123259 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="de886e71-0319-4c22-9df8-036f43e8072c" containerName="revoke-keys" Oct 30 07:55:51.123387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123273 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73d760a-cdbb-4b6a-a894-ee593c427522" containerName="revoke-keys" Oct 30 07:55:51.123387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.123282 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="860fba1b-d47b-46bb-85a5-48b1536660a6" containerName="main" Oct 30 07:55:51.127099 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.127082 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.145003 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.144975 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:55:51.149164 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.149137 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm"] Oct 30 07:55:51.164678 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.158884 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:55:51.214471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.214433 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.214742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.214601 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fh2l\" (UniqueName: \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.214977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.214864 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.316507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.316465 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.316707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.316532 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.316707 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.316619 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fh2l\" (UniqueName: \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.317341 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.317316 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.318999 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.318971 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.361197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.361166 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fh2l\" (UniqueName: \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\") pod \"maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.428922 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.428825 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:51.445309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.445266 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:51.775130 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:55:51.775092 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1440b4f9_0557_4c15_ac8b_7f53feadacf3.slice/crio-ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68 WatchSource:0}: Error finding container ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68: Status 404 returned error can't find the container with id ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68 Oct 30 07:55:51.777331 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:51.777295 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm"] Oct 30 07:55:52.399343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:52.399307 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:55:52.741073 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:52.741028 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" event={"ID":"1440b4f9-0557-4c15-ac8b-7f53feadacf3","Type":"ContainerStarted","Data":"643b1fb67d07ba112640ea2f4ddb2c00228bb5bc26c1677b699ae74c25edeed8"} Oct 30 07:55:52.741289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:52.741094 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" event={"ID":"1440b4f9-0557-4c15-ac8b-7f53feadacf3","Type":"ContainerStarted","Data":"ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68"} Oct 30 07:55:52.785498 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:52.785441 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" podStartSLOduration=1.785427321 podStartE2EDuration="1.785427321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:55:52.781980832 +0000 UTC m=+3516.472898270" watchObservedRunningTime="2026-10-30 07:55:52.785427321 +0000 UTC m=+3516.476344776" Oct 30 07:55:53.116534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:53.116493 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:53.751945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:53.751903 5894 generic.go:356] "Generic (PLEG): container finished" podID="1440b4f9-0557-4c15-ac8b-7f53feadacf3" containerID="643b1fb67d07ba112640ea2f4ddb2c00228bb5bc26c1677b699ae74c25edeed8" exitCode=0 Oct 30 07:55:53.752141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:53.751991 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" event={"ID":"1440b4f9-0557-4c15-ac8b-7f53feadacf3","Type":"ContainerDied","Data":"643b1fb67d07ba112640ea2f4ddb2c00228bb5bc26c1677b699ae74c25edeed8"} Oct 30 07:55:54.818232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:54.811769 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:54.958117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:54.958091 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:55.075487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.075452 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\" (UniqueName: \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\") pod \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " Oct 30 07:55:55.075694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.075529 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\") pod \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " Oct 30 07:55:55.075787 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.075707 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\") pod \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\" (UID: \"1440b4f9-0557-4c15-ac8b-7f53feadacf3\") " Oct 30 07:55:55.076006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.075968 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca" pod "1440b4f9-0557-4c15-ac8b-7f53feadacf3" (UID: "1440b4f9-0557-4c15-ac8b-7f53feadacf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:55:55.076390 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.076366 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:55.077805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.077783 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca" pod "1440b4f9-0557-4c15-ac8b-7f53feadacf3" (UID: "1440b4f9-0557-4c15-ac8b-7f53feadacf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:55:55.077899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.077842 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l" pod "1440b4f9-0557-4c15-ac8b-7f53feadacf3" (UID: "1440b4f9-0557-4c15-ac8b-7f53feadacf3"). InnerVolumeSpecName "kube-api-access-6fh2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:55:55.177767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.177732 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fh2l\" (UniqueName: \"kubernetes.io/projected/1440b4f9-0557-4c15-ac8b-7f53feadacf3-kube-api-access-6fh2l\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:55.177767 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.177760 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1440b4f9-0557-4c15-ac8b-7f53feadacf3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:55:55.769130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.769090 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" Oct 30 07:55:55.769321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.769129 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm" event={"ID":"1440b4f9-0557-4c15-ac8b-7f53feadacf3","Type":"ContainerDied","Data":"ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68"} Oct 30 07:55:55.769321 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:55.769186 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba9040216f5876cdd6414a1aca21026d00f2dc18974f471862a4d7134feaec68" Oct 30 07:55:56.405234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:56.405198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:58.003661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:58.003595 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:55:59.834969 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:55:59.834938 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:01.426069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:01.426035 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:01.591896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:01.591856 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:56:01.624502 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:01.624459 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:56:01.748762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:01.748668 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:01.861052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:01.861011 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:02.399739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:02.399695 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:03.207706 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.207669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:03.471541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.471511 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:56:03.516686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.516651 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\") pod \"56e594df-ee57-4057-872a-6db795d02a0e\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " Oct 30 07:56:03.517068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.517035 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume" pod "56e594df-ee57-4057-872a-6db795d02a0e" (UID: "56e594df-ee57-4057-872a-6db795d02a0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:03.517225 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.517174 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\" (UniqueName: \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\") pod \"56e594df-ee57-4057-872a-6db795d02a0e\" (UID: \"56e594df-ee57-4057-872a-6db795d02a0e\") " Oct 30 07:56:03.518186 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.518162 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e594df-ee57-4057-872a-6db795d02a0e-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:03.519914 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.519889 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9" pod "56e594df-ee57-4057-872a-6db795d02a0e" (UID: "56e594df-ee57-4057-872a-6db795d02a0e"). InnerVolumeSpecName "kube-api-access-m9gc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:03.619851 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.619760 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9gc9\" (UniqueName: \"kubernetes.io/projected/56e594df-ee57-4057-872a-6db795d02a0e-kube-api-access-m9gc9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:03.834054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.834018 5894 generic.go:356] "Generic (PLEG): container finished" podID="56e594df-ee57-4057-872a-6db795d02a0e" containerID="51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab" exitCode=0 Oct 30 07:56:03.834269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.834071 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" Oct 30 07:56:03.834269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.834104 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" event={"ID":"56e594df-ee57-4057-872a-6db795d02a0e","Type":"ContainerDied","Data":"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab"} Oct 30 07:56:03.834269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.834165 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6" event={"ID":"56e594df-ee57-4057-872a-6db795d02a0e","Type":"ContainerDied","Data":"09e4956dbe8428c345ef49751be2d31d666493ca47601a9ab2dd83d0cdfad7e3"} Oct 30 07:56:03.834269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.834188 5894 scope.go:122] "RemoveContainer" containerID="51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab" Oct 30 07:56:03.848832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.848810 5894 scope.go:122] "RemoveContainer" containerID="51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab" Oct 30 07:56:03.849136 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:03.849117 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab\": container with ID starting with 51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab not found: ID does not exist" containerID="51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab" Oct 30 07:56:03.849196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.849146 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab"} err="failed to get container status \"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab\": rpc error: code = NotFound desc = could not find container \"51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab\": container with ID starting with 51691f6ff7b5b4515c2100b7c6237b103dff3f164c2a55da7ad7ba9776935dab not found: ID does not exist" Oct 30 07:56:03.888333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.888247 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:56:03.892658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:03.892617 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f6c2-5cfdc8f7cf-lt4t6"] Oct 30 07:56:04.434196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.434155 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e594df-ee57-4057-872a-6db795d02a0e" path="/var/lib/kubelet/pods/56e594df-ee57-4057-872a-6db795d02a0e/volumes" Oct 30 07:56:04.499750 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.499716 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 07:56:04.500804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.500774 5894 cpu_manager.go:414] "containerMap: removing container" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" Oct 30 07:56:04.500804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.500799 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" Oct 30 07:56:04.501061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.500877 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1440b4f9-0557-4c15-ac8b-7f53feadacf3" containerName="revoke-keys" Oct 30 07:56:04.501061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.500887 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1440b4f9-0557-4c15-ac8b-7f53feadacf3" containerName="revoke-keys" Oct 30 07:56:04.501061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.501017 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1440b4f9-0557-4c15-ac8b-7f53feadacf3" containerName="revoke-keys" Oct 30 07:56:04.501061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.501032 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e594df-ee57-4057-872a-6db795d02a0e" containerName="payload-pre-processing" Oct 30 07:56:04.506386 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.506362 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.529267 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.529240 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 07:56:04.545683 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.544361 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 07:56:04.550852 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.550749 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.565876 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.565845 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 07:56:04.634491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634450 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.634491 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634495 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.634756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634518 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.634756 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634702 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.634881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634770 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.634943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634869 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.634943 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634921 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.634995 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hhn\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.635094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635067 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.635163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635103 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.635233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635215 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635248 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635307 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.635363 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635335 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635448 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635364 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635482 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx54q\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635534 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635525 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.635645 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.635614 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737357 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.737446 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737449 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737468 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737490 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737505 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737522 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737597 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx54q\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737688 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.737760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737753 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737822 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737902 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.737949 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738034 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738035 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738079 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738119 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738208 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738152 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hhn\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738297 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738314 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738419 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738444 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738030 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.738764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738551 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.739066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738773 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.739066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738865 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.739066 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.738878 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.740480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.740451 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.740602 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.740566 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.740805 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.740783 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.741257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.741240 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.764112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.764080 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.764112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.764102 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.764419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.764394 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hhn\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\") pod \"maas-default-gateway-openshift-default-6dcf757899-767jp\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:04.766994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.766967 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx54q\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\") pod \"maas-default-gateway-openshift-default-6dcf757899-v7mpl\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.823485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.823447 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:04.830172 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:04.829408 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:56:04.878742 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:04.878708 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:05.175822 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.172489 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 07:56:05.183816 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:56:05.183766 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997beedc_265c_4542_b38e_100bdae82e19.slice/crio-b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa WatchSource:0}: Error finding container b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa: Status 404 returned error can't find the container with id b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa Oct 30 07:56:05.225823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.207200 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.225823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.224733 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.225823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.224794 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.225823 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.214156 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:05.226974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.226741 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 07:56:05.242028 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:56:05.241993 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d37e1d_1a39_4f9c_8f4f_9163bd1197a3.slice/crio-86ce2226bea4b4c00b683a7795603060d87cf9b9845ae46ed8d75b4e13eac4d3 WatchSource:0}: Error finding container 86ce2226bea4b4c00b683a7795603060d87cf9b9845ae46ed8d75b4e13eac4d3: Status 404 returned error can't find the container with id 86ce2226bea4b4c00b683a7795603060d87cf9b9845ae46ed8d75b4e13eac4d3 Oct 30 07:56:05.245782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.245741 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.245883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.245819 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.245883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.245852 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 07:56:05.761978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.761933 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" podUID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" containerName="main" containerID="cri-o://c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc" gracePeriod=45 Oct 30 07:56:05.860970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.860930 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" event={"ID":"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3","Type":"ContainerStarted","Data":"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15"} Oct 30 07:56:05.863314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.863274 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" event={"ID":"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3","Type":"ContainerStarted","Data":"86ce2226bea4b4c00b683a7795603060d87cf9b9845ae46ed8d75b4e13eac4d3"} Oct 30 07:56:05.868241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.868206 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" event={"ID":"997beedc-265c-4542-b38e-100bdae82e19","Type":"ContainerStarted","Data":"d34b6a0baed90b366cd5f93bb8d8f0d830e18da4535875e9be911038f3b26fd2"} Oct 30 07:56:05.868379 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.868271 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" event={"ID":"997beedc-265c-4542-b38e-100bdae82e19","Type":"ContainerStarted","Data":"b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa"} Oct 30 07:56:05.879346 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.879322 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:05.927866 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:05.927803 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podStartSLOduration=1.9277873159999999 podStartE2EDuration="1.927787316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:05.921671051 +0000 UTC m=+3529.612588496" watchObservedRunningTime="2026-10-30 07:56:05.927787316 +0000 UTC m=+3529.618704752" Oct 30 07:56:06.159659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.156929 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:56:06.215660 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.215570 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podStartSLOduration=2.215546393 podStartE2EDuration="2.215546393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:05.984881852 +0000 UTC m=+3529.675799290" watchObservedRunningTime="2026-10-30 07:56:06.215546393 +0000 UTC m=+3529.906463832" Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289197 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289287 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289503 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289757 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\" (UniqueName: \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.289828 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\") pod \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\" (UID: \"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9\") " Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.291186 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:06.307665 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.297233 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:06.346959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.325494 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.363902 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "kube-api-access-sbl7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.366434 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.372924 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs" pod "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" (UID: "e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404111 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404156 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404172 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404185 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404250 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbl7k\" (UniqueName: \"kubernetes.io/projected/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-kube-api-access-sbl7k\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.466662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.404263 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:06.604723 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.603700 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" containerID="cri-o://d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366" gracePeriod=25 Oct 30 07:56:06.627932 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.625773 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" containerID="cri-o://fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed" gracePeriod=25 Oct 30 07:56:06.824098 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.824014 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:06.829142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.828769 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:06.829142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.828836 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:06.890627 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.890529 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:06.890928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.890895 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:06.918793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.918753 5894 generic.go:356] "Generic (PLEG): container finished" podID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" containerID="c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc" exitCode=0 Oct 30 07:56:06.929175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.920043 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" Oct 30 07:56:06.929175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.923807 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" event={"ID":"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9","Type":"ContainerDied","Data":"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc"} Oct 30 07:56:06.929175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.928817 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr" event={"ID":"e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9","Type":"ContainerDied","Data":"4531b8ce1e48c9f5806a1a6ab4817b13f5c9d51a59c6a68ec559a477e564ccfd"} Oct 30 07:56:06.929175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.928850 5894 scope.go:122] "RemoveContainer" containerID="c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc" Oct 30 07:56:06.946915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.946883 5894 scope.go:122] "RemoveContainer" containerID="c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc" Oct 30 07:56:06.947274 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:06.947250 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc\": container with ID starting with c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc not found: ID does not exist" containerID="c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc" Oct 30 07:56:06.947349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:06.947283 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc"} err="failed to get container status \"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc\": rpc error: code = NotFound desc = could not find container \"c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc\": container with ID starting with c92f2f266072debd3b113c0b352010999a2f08e3157ac6f8a5b085316d0fb3cc not found: ID does not exist" Oct 30 07:56:07.019716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.003692 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:56:07.028235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.028189 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-759ae6/test-model-759ae6-kserve-558db6dccc-wpfkr"] Oct 30 07:56:07.133781 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.131183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:07.824147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.824106 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:07.824799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.824758 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:07.892659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.890143 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:07.892659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:07.890209 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:08.437779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:08.437740 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" path="/var/lib/kubelet/pods/e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9/volumes" Oct 30 07:56:08.825686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:08.824137 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:08.825686 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:08.824201 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:08.902671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:08.886739 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:08.902671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:08.886804 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:09.116714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:09.115797 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:09.824611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:09.824571 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:09.824856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:09.824655 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:09.894666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:09.894198 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:09.894666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:09.894260 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:10.824565 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:10.824498 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:10.824804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:10.824561 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:10.880116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:10.880078 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:10.880411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:10.880385 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:11.044197 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:11.044159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:11.824930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:11.824885 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:11.825135 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:11.824969 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:11.879605 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:11.879569 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:11.879795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:11.879651 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:12.303986 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.303943 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:12.304410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.303956 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:12.329402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.329365 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8"] Oct 30 07:56:12.330782 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.330757 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" containerName="main" Oct 30 07:56:12.330882 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.330784 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" containerName="main" Oct 30 07:56:12.331045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.331022 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ab6e2d-08c2-4b2e-8992-5ca0e6df46b9" containerName="main" Oct 30 07:56:12.337569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.337545 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.355541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.355503 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:56:12.355848 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.355821 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8"] Oct 30 07:56:12.360989 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.360942 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:56:12.407961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.407925 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.408273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.408245 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tn4\" (UniqueName: \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.408347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.408309 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.509942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.509903 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.510147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.510127 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tn4\" (UniqueName: \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.510214 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.510175 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.510792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.510771 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.512903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.512874 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.534226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.534169 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tn4\" (UniqueName: \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\") pod \"maas-api-revoke-sub-a4af27a0-pq8z8\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.659269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.659181 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:12.825181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.824677 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-v7mpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" start-of-body= Oct 30 07:56:12.825181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.824734 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.103:15021/healthz/ready\": dial tcp 10.134.1.103:15021: connect: connection refused" Oct 30 07:56:12.839796 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.839754 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:12.892212 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.892159 5894 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6dcf757899-767jp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" start-of-body= Oct 30 07:56:12.893444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:12.893026 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.104:15021/healthz/ready\": dial tcp 10.134.1.104:15021: connect: connection refused" Oct 30 07:56:12.996675 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:56:12.996611 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc67126_aa1c_4504_a7f1_b4f361af286f.slice/crio-72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c WatchSource:0}: Error finding container 72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c: Status 404 returned error can't find the container with id 72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c Oct 30 07:56:13.000269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:13.000222 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8"] Oct 30 07:56:13.994273 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:13.994231 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" event={"ID":"9bc67126-aa1c-4504-a7f1-b4f361af286f","Type":"ContainerStarted","Data":"8c9931a7df0edf828a239e3bf5535cd7a46b53051bfdb3ba08ecb2151ce96d9a"} Oct 30 07:56:13.994941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:13.994911 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" event={"ID":"9bc67126-aa1c-4504-a7f1-b4f361af286f","Type":"ContainerStarted","Data":"72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c"} Oct 30 07:56:14.041661 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.038285 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" podStartSLOduration=2.038263864 podStartE2EDuration="2.038263864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:56:14.035355898 +0000 UTC m=+3537.726273339" watchObservedRunningTime="2026-10-30 07:56:14.038263864 +0000 UTC m=+3537.729181301" Oct 30 07:56:14.340457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.340425 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:14.341460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.341431 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:14.342142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.342112 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 07:56:14.403482 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.403450 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:14.403946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.403925 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:14.404776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.404752 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 07:56:14.550825 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:14.550792 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:15.005391 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:15.005351 5894 generic.go:356] "Generic (PLEG): container finished" podID="9bc67126-aa1c-4504-a7f1-b4f361af286f" containerID="8c9931a7df0edf828a239e3bf5535cd7a46b53051bfdb3ba08ecb2151ce96d9a" exitCode=0 Oct 30 07:56:15.005915 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:15.005441 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" event={"ID":"9bc67126-aa1c-4504-a7f1-b4f361af286f","Type":"ContainerDied","Data":"8c9931a7df0edf828a239e3bf5535cd7a46b53051bfdb3ba08ecb2151ce96d9a"} Oct 30 07:56:16.185434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.185408 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:16.259901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.259864 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:16.283923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.283884 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\" (UniqueName: \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\") pod \"9bc67126-aa1c-4504-a7f1-b4f361af286f\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " Oct 30 07:56:16.284400 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.284368 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\") pod \"9bc67126-aa1c-4504-a7f1-b4f361af286f\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " Oct 30 07:56:16.284559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.284489 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\") pod \"9bc67126-aa1c-4504-a7f1-b4f361af286f\" (UID: \"9bc67126-aa1c-4504-a7f1-b4f361af286f\") " Oct 30 07:56:16.300976 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.285462 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca" pod "9bc67126-aa1c-4504-a7f1-b4f361af286f" (UID: "9bc67126-aa1c-4504-a7f1-b4f361af286f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:16.302059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.301999 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4" pod "9bc67126-aa1c-4504-a7f1-b4f361af286f" (UID: "9bc67126-aa1c-4504-a7f1-b4f361af286f"). InnerVolumeSpecName "kube-api-access-99tn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:16.306216 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.306154 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca" pod "9bc67126-aa1c-4504-a7f1-b4f361af286f" (UID: "9bc67126-aa1c-4504-a7f1-b4f361af286f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:56:16.386909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.386874 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:16.386909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.386903 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bc67126-aa1c-4504-a7f1-b4f361af286f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:16.386909 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:16.386913 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tn4\" (UniqueName: \"kubernetes.io/projected/9bc67126-aa1c-4504-a7f1-b4f361af286f-kube-api-access-99tn4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:17.023333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:17.023299 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" event={"ID":"9bc67126-aa1c-4504-a7f1-b4f361af286f","Type":"ContainerDied","Data":"72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c"} Oct 30 07:56:17.023509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:17.023342 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8" Oct 30 07:56:17.023509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:17.023350 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72313d6e615df7d3390509b888442adcd6540b671b31d405309dc69c73c2315c" Oct 30 07:56:17.937505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:17.937471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:19.528701 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:19.528627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:21.272409 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:21.272327 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:22.303788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:22.303751 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:22.304210 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:22.303840 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:56:22.761618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:22.761536 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:22.922858 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:22.922823 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:24.610809 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:24.610768 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:26.445305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:26.445265 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:26.952799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:26.952728 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:56:26.989192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:26.989163 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:56:27.043343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.043295 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\" (UniqueName: \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\") pod \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " Oct 30 07:56:27.043568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.043396 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\") pod \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\" (UID: \"a9d5f6ca-4068-4c97-bbef-18b53675ff62\") " Oct 30 07:56:27.043568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.043467 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\" (UniqueName: \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\") pod \"542c41c5-8a38-41c3-824f-025144acfc4f\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " Oct 30 07:56:27.043568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.043554 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\") pod \"542c41c5-8a38-41c3-824f-025144acfc4f\" (UID: \"542c41c5-8a38-41c3-824f-025144acfc4f\") " Oct 30 07:56:27.044417 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.044393 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume" pod "542c41c5-8a38-41c3-824f-025144acfc4f" (UID: "542c41c5-8a38-41c3-824f-025144acfc4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:27.044961 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.044935 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume" pod "a9d5f6ca-4068-4c97-bbef-18b53675ff62" (UID: "a9d5f6ca-4068-4c97-bbef-18b53675ff62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:27.046520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.046491 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm" pod "a9d5f6ca-4068-4c97-bbef-18b53675ff62" (UID: "a9d5f6ca-4068-4c97-bbef-18b53675ff62"). InnerVolumeSpecName "kube-api-access-sm7wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:27.046911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.046892 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s" pod "542c41c5-8a38-41c3-824f-025144acfc4f" (UID: "542c41c5-8a38-41c3-824f-025144acfc4f"). InnerVolumeSpecName "kube-api-access-zrk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:27.114467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.114431 5894 generic.go:356] "Generic (PLEG): container finished" podID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerID="fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed" exitCode=0 Oct 30 07:56:27.114702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.114478 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" Oct 30 07:56:27.114702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.114512 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" event={"ID":"a9d5f6ca-4068-4c97-bbef-18b53675ff62","Type":"ContainerDied","Data":"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed"} Oct 30 07:56:27.114702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.114566 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk" event={"ID":"a9d5f6ca-4068-4c97-bbef-18b53675ff62","Type":"ContainerDied","Data":"d1556cd895c579e50222962e4f53d2cd3c9d713cce559950c0bcfe0d8087e2d3"} Oct 30 07:56:27.114702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.114593 5894 scope.go:122] "RemoveContainer" containerID="fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed" Oct 30 07:56:27.116332 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.116305 5894 generic.go:356] "Generic (PLEG): container finished" podID="542c41c5-8a38-41c3-824f-025144acfc4f" containerID="d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366" exitCode=0 Oct 30 07:56:27.116444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.116334 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" event={"ID":"542c41c5-8a38-41c3-824f-025144acfc4f","Type":"ContainerDied","Data":"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366"} Oct 30 07:56:27.116444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.116412 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" event={"ID":"542c41c5-8a38-41c3-824f-025144acfc4f","Type":"ContainerDied","Data":"4a2165deeaf4fdeaa5f302cc98ce595c582830598079966ead3ce5ba071455e7"} Oct 30 07:56:27.116444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.116355 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8" Oct 30 07:56:27.126222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.126202 5894 scope.go:122] "RemoveContainer" containerID="fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed" Oct 30 07:56:27.126728 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:27.126516 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed\": container with ID starting with fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed not found: ID does not exist" containerID="fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed" Oct 30 07:56:27.126728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.126545 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed"} err="failed to get container status \"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed\": rpc error: code = NotFound desc = could not find container \"fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed\": container with ID starting with fd37d44a29de23ede8312c729c5008885afec449991ea76c3fd166aa3d5c47ed not found: ID does not exist" Oct 30 07:56:27.126728 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.126562 5894 scope.go:122] "RemoveContainer" containerID="d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366" Oct 30 07:56:27.137309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.137292 5894 scope.go:122] "RemoveContainer" containerID="d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366" Oct 30 07:56:27.137592 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:27.137567 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366\": container with ID starting with d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366 not found: ID does not exist" containerID="d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366" Oct 30 07:56:27.137684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.137598 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366"} err="failed to get container status \"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366\": rpc error: code = NotFound desc = could not find container \"d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366\": container with ID starting with d698ead4ff97622b60cd6568e36c570cb00ce394b3e5b1d83e0ae16838147366 not found: ID does not exist" Oct 30 07:56:27.145151 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.145131 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm7wm\" (UniqueName: \"kubernetes.io/projected/a9d5f6ca-4068-4c97-bbef-18b53675ff62-kube-api-access-sm7wm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:27.145236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.145153 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9d5f6ca-4068-4c97-bbef-18b53675ff62-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:27.145236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.145165 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrk9s\" (UniqueName: \"kubernetes.io/projected/542c41c5-8a38-41c3-824f-025144acfc4f-kube-api-access-zrk9s\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:27.145236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.145175 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/542c41c5-8a38-41c3-824f-025144acfc4f-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:27.168834 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.168807 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:56:27.173306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.173281 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc3a23-d8f54476d-tgsp8"] Oct 30 07:56:27.207025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.206947 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:56:27.211589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:27.211566 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc3a23-56c968c9c5-n7tdk"] Oct 30 07:56:28.117671 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:28.117624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:28.433978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:28.433899 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" path="/var/lib/kubelet/pods/542c41c5-8a38-41c3-824f-025144acfc4f/volumes" Oct 30 07:56:28.434383 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:28.434369 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" path="/var/lib/kubelet/pods/a9d5f6ca-4068-4c97-bbef-18b53675ff62/volumes" Oct 30 07:56:28.715147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:28.715045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:29.505865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:29.505829 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:30.090101 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.090056 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:30.213511 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.213474 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9"] Oct 30 07:56:30.216412 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.216382 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:30.225287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.225259 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-61f6c2-nfbq9"] Oct 30 07:56:30.348625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.348533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:30.433790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.433758 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de886e71-0319-4c22-9df8-036f43e8072c" path="/var/lib/kubelet/pods/de886e71-0319-4c22-9df8-036f43e8072c/volumes" Oct 30 07:56:30.760289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:30.760201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:32.134194 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:32.134154 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:32.451724 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:32.451627 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:56:32.534213 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:32.534179 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:33.883765 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:33.883725 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:34.830859 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:34.830819 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:56:35.814815 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:35.814778 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:37.143236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:37.143200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:37.276700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:37.276622 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:37.548140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:37.548105 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:37.766626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:37.766585 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:38.563748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:38.563715 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:38.683845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:38.683803 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:38.814126 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:38.814046 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:39.306883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:39.306848 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:39.470259 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:39.470223 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:41.026053 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:41.026014 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:41.710030 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:41.709991 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:42.702429 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:42.702390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:44.439433 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:44.439392 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:45.107649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:45.107595 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:56:45.107889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:45.107853 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" containerID="cri-o://3c4b2c2d0e2554df8a693ab5147e63c92f222a2c95c20822fc82988d119094cc" gracePeriod=30 Oct 30 07:56:45.774354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:45.774318 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:56:45.774798 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:45.774519 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" containerID="cri-o://a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6" gracePeriod=30 Oct 30 07:56:46.201990 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:46.201957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:47.110269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:47.110232 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 07:56:47.110734 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:47.110323 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 07:56:47.852970 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:47.852932 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:49.536027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:49.535991 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:50.366576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.366301 5894 generic.go:356] "Generic (PLEG): container finished" podID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerID="3c4b2c2d0e2554df8a693ab5147e63c92f222a2c95c20822fc82988d119094cc" exitCode=0 Oct 30 07:56:50.366576 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.366337 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" event={"ID":"83d1cfb5-9803-4356-995b-815e0dbdef5c","Type":"ContainerDied","Data":"3c4b2c2d0e2554df8a693ab5147e63c92f222a2c95c20822fc82988d119094cc"} Oct 30 07:56:50.401070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.401041 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:56:50.525271 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525231 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525281 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525334 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525452 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525398 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525461 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525510 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525537 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525582 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525711 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:50.525996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525739 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:50.525996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525773 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:50.525996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525917 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\") pod \"83d1cfb5-9803-4356-995b-815e0dbdef5c\" (UID: \"83d1cfb5-9803-4356-995b-815e0dbdef5c\") " Oct 30 07:56:50.525996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.525928 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:50.526272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526159 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:50.526790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526755 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.526790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526789 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.526925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526805 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.526925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526819 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83d1cfb5-9803-4356-995b-815e0dbdef5c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.526925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.526835 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.528027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.528005 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "kube-api-access-lhrmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:50.528232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.528209 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:50.528232 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.528224 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:56:50.528342 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.528234 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token" pod "83d1cfb5-9803-4356-995b-815e0dbdef5c" (UID: "83d1cfb5-9803-4356-995b-815e0dbdef5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:50.627903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.627812 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.627903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.627841 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.627903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.627850 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrmz\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-kube-api-access-lhrmz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:50.627903 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:50.627858 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83d1cfb5-9803-4356-995b-815e0dbdef5c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.041881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.041854 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:56:51.126207 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.126169 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134360 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134424 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134491 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134536 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134738 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134790 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134881 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.134999 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.135979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.135055 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\") pod \"9acb7928-044a-4865-b485-8d162cf734e3\" (UID: \"9acb7928-044a-4865-b485-8d162cf734e3\") " Oct 30 07:56:51.137732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.137471 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:51.137732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.137523 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:51.137732 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.137600 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:51.138111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.138046 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:51.138111 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.138069 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:56:51.144085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.144048 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "kube-api-access-mtjt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:51.144242 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.144114 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:56:51.145130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.145093 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:56:51.146560 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.146526 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo" pod "9acb7928-044a-4865-b485-8d162cf734e3" (UID: "9acb7928-044a-4865-b485-8d162cf734e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:56:51.237261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237218 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9acb7928-044a-4865-b485-8d162cf734e3-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237253 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237261 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237266 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237280 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237291 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237302 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9acb7928-044a-4865-b485-8d162cf734e3-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237313 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjt5\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-kube-api-access-mtjt5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237324 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9acb7928-044a-4865-b485-8d162cf734e3-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.237521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.237335 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9acb7928-044a-4865-b485-8d162cf734e3-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:56:51.406027 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.405913 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" event={"ID":"83d1cfb5-9803-4356-995b-815e0dbdef5c","Type":"ContainerDied","Data":"85c4184e9ea1730117b19d21b776086afe1cfada9317283b9b6ee81cef50f609"} Oct 30 07:56:51.406370 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.406269 5894 scope.go:122] "RemoveContainer" containerID="3c4b2c2d0e2554df8a693ab5147e63c92f222a2c95c20822fc82988d119094cc" Oct 30 07:56:51.406702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.406684 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw" Oct 30 07:56:51.433716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.433613 5894 generic.go:356] "Generic (PLEG): container finished" podID="9acb7928-044a-4865-b485-8d162cf734e3" containerID="a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6" exitCode=0 Oct 30 07:56:51.434010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.433996 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" Oct 30 07:56:51.437890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.437768 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" event={"ID":"9acb7928-044a-4865-b485-8d162cf734e3","Type":"ContainerDied","Data":"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6"} Oct 30 07:56:51.437890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.437840 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct" event={"ID":"9acb7928-044a-4865-b485-8d162cf734e3","Type":"ContainerDied","Data":"6ec6ee8f37fe4c3838176bd6fd43dbb98a993156c65de218e22eb001e8fe5bb4"} Oct 30 07:56:51.474061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.466816 5894 scope.go:122] "RemoveContainer" containerID="a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6" Oct 30 07:56:51.499455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.499123 5894 scope.go:122] "RemoveContainer" containerID="a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6" Oct 30 07:56:51.500905 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:56:51.500714 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6\": container with ID starting with a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6 not found: ID does not exist" containerID="a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6" Oct 30 07:56:51.500905 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.500755 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6"} err="failed to get container status \"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6\": rpc error: code = NotFound desc = could not find container \"a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6\": container with ID starting with a98f8647fdeafc4dba7292659214c88315c4446892d40d4fe4f572e767473cc6 not found: ID does not exist" Oct 30 07:56:51.525343 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.525260 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:56:51.537174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.537091 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-94rvw"] Oct 30 07:56:51.573158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.573117 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:56:51.575600 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.575570 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-vh8ct"] Oct 30 07:56:51.677496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.677400 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:51.810170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.810127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:51.849558 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.848761 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm"] Oct 30 07:56:51.852772 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:51.852510 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fc3a23-49dcm"] Oct 30 07:56:52.367622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:52.367590 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:52.435095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:52.435052 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1440b4f9-0557-4c15-ac8b-7f53feadacf3" path="/var/lib/kubelet/pods/1440b4f9-0557-4c15-ac8b-7f53feadacf3/volumes" Oct 30 07:56:52.435829 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:52.435810 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" path="/var/lib/kubelet/pods/83d1cfb5-9803-4356-995b-815e0dbdef5c/volumes" Oct 30 07:56:52.436485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:52.436468 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9acb7928-044a-4865-b485-8d162cf734e3" path="/var/lib/kubelet/pods/9acb7928-044a-4865-b485-8d162cf734e3/volumes" Oct 30 07:56:53.598060 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:53.598022 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:55.235777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:55.235743 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:56.913972 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:56.913927 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:58.656010 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:58.655970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:59.301071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:59.301036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:59.418514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:59.418474 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:56:59.666347 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:56:59.666237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:00.315064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315027 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6"] Oct 30 07:57:00.315832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315817 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" Oct 30 07:57:00.315895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315834 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" Oct 30 07:57:00.315895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315843 5894 cpu_manager.go:414] "containerMap: removing container" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" Oct 30 07:57:00.315895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315849 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" Oct 30 07:57:00.315895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315864 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" Oct 30 07:57:00.315895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315869 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315916 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9bc67126-aa1c-4504-a7f1-b4f361af286f" containerName="revoke-keys" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315921 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bc67126-aa1c-4504-a7f1-b4f361af286f" containerName="revoke-keys" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315930 5894 cpu_manager.go:414] "containerMap: removing container" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.315936 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.316045 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bc67126-aa1c-4504-a7f1-b4f361af286f" containerName="revoke-keys" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.316053 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d1cfb5-9803-4356-995b-815e0dbdef5c" containerName="istio-proxy" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.316063 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d5f6ca-4068-4c97-bbef-18b53675ff62" containerName="payload-pre-processing" Oct 30 07:57:00.316071 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.316071 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="542c41c5-8a38-41c3-824f-025144acfc4f" containerName="payload-processing" Oct 30 07:57:00.316311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.316078 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9acb7928-044a-4865-b485-8d162cf734e3" containerName="istio-proxy" Oct 30 07:57:00.321814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.321791 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.338430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.338398 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:57:00.339150 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.339130 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6"] Oct 30 07:57:00.347333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.347300 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:57:00.394272 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.394236 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.394459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.394437 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9pw\" (UniqueName: \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.394564 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.394542 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.456173 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.456123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:00.498444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.498401 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.500313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.500280 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.501774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.501624 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.502196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.502168 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9pw\" (UniqueName: \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.533664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.516109 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.560983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.560951 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9pw\" (UniqueName: \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\") pod \"maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.638398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.638303 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:00.947090 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:57:00.947049 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22e9b00_cfd8_450d_bcc5_ec573a14f9ac.slice/crio-a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c WatchSource:0}: Error finding container a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c: Status 404 returned error can't find the container with id a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c Oct 30 07:57:00.949493 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:00.949462 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6"] Oct 30 07:57:01.570076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:01.569970 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" event={"ID":"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac","Type":"ContainerStarted","Data":"a12697058a37a35ec2b46b6dcbe5afaa83b69727c4b21c3ed0376b7b514946e3"} Oct 30 07:57:01.570076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:01.570037 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" event={"ID":"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac","Type":"ContainerStarted","Data":"a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c"} Oct 30 07:57:01.613125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:01.613062 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" podStartSLOduration=1.613047438 podStartE2EDuration="1.613047438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:57:01.610418025 +0000 UTC m=+3585.301335466" watchObservedRunningTime="2026-10-30 07:57:01.613047438 +0000 UTC m=+3585.303964876" Oct 30 07:57:02.337389 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:02.337346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:02.579789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:02.579752 5894 generic.go:356] "Generic (PLEG): container finished" podID="f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" containerID="a12697058a37a35ec2b46b6dcbe5afaa83b69727c4b21c3ed0376b7b514946e3" exitCode=0 Oct 30 07:57:02.579973 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:02.579842 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" event={"ID":"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac","Type":"ContainerDied","Data":"a12697058a37a35ec2b46b6dcbe5afaa83b69727c4b21c3ed0376b7b514946e3"} Oct 30 07:57:03.769279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.769242 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:03.846917 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.846817 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\") pod \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " Oct 30 07:57:03.847103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.846935 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\") pod \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " Oct 30 07:57:03.847103 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.847010 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\" (UniqueName: \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\") pod \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\" (UID: \"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac\") " Oct 30 07:57:03.847419 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.847385 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca" pod "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" (UID: "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:03.847994 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.847968 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.849317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.849292 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw" pod "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" (UID: "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac"). InnerVolumeSpecName "kube-api-access-5m9pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:03.849672 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.849618 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca" pod "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" (UID: "f22e9b00-cfd8-450d-bcc5-ec573a14f9ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:57:03.949170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.949136 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9pw\" (UniqueName: \"kubernetes.io/projected/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-kube-api-access-5m9pw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.949170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.949168 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:03.976456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:03.976422 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:04.596521 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:04.596488 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" Oct 30 07:57:04.596759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:04.596525 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6" event={"ID":"f22e9b00-cfd8-450d-bcc5-ec573a14f9ac","Type":"ContainerDied","Data":"a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c"} Oct 30 07:57:04.596759 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:04.596576 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a5205df70b46174f39736bd3ddd36e1d22ac3480753852bebbe2a300972a3c" Oct 30 07:57:04.830392 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:57:04.830357 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:57:05.759048 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:05.759006 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:05.890947 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:05.890911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:07.357456 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:07.357398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:08.985192 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:08.985153 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:10.646217 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:10.646179 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:10.835533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:10.835492 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:57:10.868916 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:10.868884 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:57:10.988595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:10.988475 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:11.109569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:11.109533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:12.030411 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:12.030378 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84"] Oct 30 07:57:12.042743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:12.042715 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93702fbb-mbg84"] Oct 30 07:57:12.365153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:12.365065 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:12.453863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:12.453810 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cd6bedb-1158-401b-b376-2e8eda058a7d" path="/var/lib/kubelet/pods/7cd6bedb-1158-401b-b376-2e8eda058a7d/volumes" Oct 30 07:57:12.486220 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:12.486187 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:14.160928 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:14.160893 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:15.835890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:15.835842 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" containerID="cri-o://3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97" gracePeriod=25 Oct 30 07:57:15.869800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:15.869755 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" containerID="cri-o://1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e" gracePeriod=25 Oct 30 07:57:16.067704 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:16.067661 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:17.934043 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:17.933995 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:19.551505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:19.551465 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:21.139507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:21.139476 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:22.737256 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:22.737210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:23.031430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:23.031394 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw"] Oct 30 07:57:23.043611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:23.043583 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ce4318e-mvbtw"] Oct 30 07:57:23.857542 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:23.857505 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:57:24.334812 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:24.334780 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:24.434777 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:24.434746 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7" path="/var/lib/kubelet/pods/577a4ff7-cfca-4739-8a2e-b7a20b3e6bc7/volumes" Oct 30 07:57:24.867300 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:24.867260 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:57:26.255268 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:26.255237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:27.994748 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:27.994612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:29.520550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:29.520515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:31.153620 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:31.153587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:32.644218 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:32.644183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:32.817628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:32.817593 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:33.857664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:33.857592 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:57:34.513590 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:34.513553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:34.830545 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:57:34.830505 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:57:34.866837 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:34.866805 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:57:36.092007 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.090421 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:57:36.162853 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.162823 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:57:36.203423 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.203391 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:57:36.253927 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.253876 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:36.325744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.296246 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\" (UniqueName: \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\") pod \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " Oct 30 07:57:36.325744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.296511 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\") pod \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " Oct 30 07:57:36.325744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.296710 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\") pod \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\" (UID: \"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae\") " Oct 30 07:57:36.325744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.296783 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\" (UniqueName: \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\") pod \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\" (UID: \"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb\") " Oct 30 07:57:36.325744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.323513 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w" pod "3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" (UID: "3bcd7b22-7bca-4bb3-b72f-f714e13d02fb"). InnerVolumeSpecName "kube-api-access-4q87w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:36.327685 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.327593 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume" pod "3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" (UID: "3bcd7b22-7bca-4bb3-b72f-f714e13d02fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:36.327856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.327745 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj" pod "b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" (UID: "b224b8a5-9cf6-4cc1-a379-6dcc1e565cae"). InnerVolumeSpecName "kube-api-access-t26dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:57:36.327930 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.327862 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume" pod "b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" (UID: "b224b8a5-9cf6-4cc1-a379-6dcc1e565cae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:57:36.398982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.398898 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q87w\" (UniqueName: \"kubernetes.io/projected/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-kube-api-access-4q87w\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:36.398982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.398928 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t26dj\" (UniqueName: \"kubernetes.io/projected/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-kube-api-access-t26dj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:36.398982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.398939 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:36.398982 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.398948 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:57:36.878436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.878403 5894 generic.go:356] "Generic (PLEG): container finished" podID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerID="3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97" exitCode=0 Oct 30 07:57:36.878617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.878454 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" Oct 30 07:57:36.878617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.878481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" event={"ID":"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae","Type":"ContainerDied","Data":"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97"} Oct 30 07:57:36.878617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.878534 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r" event={"ID":"b224b8a5-9cf6-4cc1-a379-6dcc1e565cae","Type":"ContainerDied","Data":"a4a0cd52d175da808434569a1d5af1250ac134ec1911754ee58ce2156d7dc00c"} Oct 30 07:57:36.878617 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.878557 5894 scope.go:122] "RemoveContainer" containerID="3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97" Oct 30 07:57:36.880095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.880073 5894 generic.go:356] "Generic (PLEG): container finished" podID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerID="1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e" exitCode=0 Oct 30 07:57:36.880257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.880114 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" Oct 30 07:57:36.880257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.880140 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" event={"ID":"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb","Type":"ContainerDied","Data":"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e"} Oct 30 07:57:36.880257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.880181 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw" event={"ID":"3bcd7b22-7bca-4bb3-b72f-f714e13d02fb","Type":"ContainerDied","Data":"d0eed0c3287e3149b8e4130b785cfa840a13fde153cd9adbbfbaaf5cc55d4f32"} Oct 30 07:57:36.891499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.891481 5894 scope.go:122] "RemoveContainer" containerID="3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97" Oct 30 07:57:36.891816 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:57:36.891794 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97\": container with ID starting with 3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97 not found: ID does not exist" containerID="3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97" Oct 30 07:57:36.891898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.891823 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97"} err="failed to get container status \"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97\": rpc error: code = NotFound desc = could not find container \"3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97\": container with ID starting with 3ae68c27f52f02dd95732f9a37580f88f1940d5adb52c1355121a5298b3fdb97 not found: ID does not exist" Oct 30 07:57:36.891898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.891841 5894 scope.go:122] "RemoveContainer" containerID="1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e" Oct 30 07:57:36.902201 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.902178 5894 scope.go:122] "RemoveContainer" containerID="1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e" Oct 30 07:57:36.902476 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:57:36.902456 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e\": container with ID starting with 1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e not found: ID does not exist" containerID="1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e" Oct 30 07:57:36.902550 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.902484 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e"} err="failed to get container status \"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e\": rpc error: code = NotFound desc = could not find container \"1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e\": container with ID starting with 1f1f14c24b017d3a91788c54f991c670279aba74f4f1081b116ddc66a3c5948e not found: ID does not exist" Oct 30 07:57:36.931648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.931608 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:57:36.935714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.935685 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-759ae6-78cc9bc9d7-cb2pw"] Oct 30 07:57:36.978372 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.978336 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:57:36.981984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:36.981958 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-759ae6-84764d9bcb-nst6r"] Oct 30 07:57:37.915792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:37.915758 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:38.433398 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:38.433361 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" path="/var/lib/kubelet/pods/3bcd7b22-7bca-4bb3-b72f-f714e13d02fb/volumes" Oct 30 07:57:38.433839 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:38.433822 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" path="/var/lib/kubelet/pods/b224b8a5-9cf6-4cc1-a379-6dcc1e565cae/volumes" Oct 30 07:57:39.648306 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:39.648271 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:41.531468 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:41.531433 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:43.252783 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:43.252746 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:44.088069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:44.088034 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:44.209394 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:44.209359 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:44.990195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:44.990157 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:46.738987 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:46.738955 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:48.406559 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:48.406521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:50.000983 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:50.000941 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:51.388810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:51.388764 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:52.033816 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.033768 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:52.148290 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.148252 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:52.162847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.162811 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6"] Oct 30 07:57:52.174566 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.174537 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-759ae6-pq4r6"] Oct 30 07:57:52.283089 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.283051 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:52.434037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.433941 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" path="/var/lib/kubelet/pods/f22e9b00-cfd8-450d-bcc5-ec573a14f9ac/volumes" Oct 30 07:57:52.679431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.679393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:52.947893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:52.947860 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:53.071721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:53.071687 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:53.991093 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:53.991056 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:55.646175 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:55.646144 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:56.097314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:56.097278 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:56.226226 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:56.226187 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:57.504500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:57.504459 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:59.023978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.023932 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:59.145305 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.145264 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:59.348703 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.348661 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:57:59.375485 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.375445 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:57:59.375841 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.375812 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" containerID="cri-o://493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78" gracePeriod=30 Oct 30 07:57:59.494435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:57:59.494380 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:00.366509 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:00.366472 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:00.503700 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:00.503665 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:01.176556 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:01.176515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:02.955307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:02.955270 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:04.544895 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.544856 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:04.718616 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.718583 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797439 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797559 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797608 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797672 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797712 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797834 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.797863 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.798229 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.798023 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\") pod \"ed868f9d-f92d-4189-a052-22bde008e5f9\" (UID: \"ed868f9d-f92d-4189-a052-22bde008e5f9\") " Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.799115 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.800527 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.801567 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.801734 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "kube-api-access-257rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.801878 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.801967 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.802058 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:58:04.808313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.802356 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:04.812662 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.812509 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:04.836688 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.836649 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy" pod "ed868f9d-f92d-4189-a052-22bde008e5f9" (UID: "ed868f9d-f92d-4189-a052-22bde008e5f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:04.846121 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:58:04.846082 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:58:04.901642 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901596 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257rm\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-kube-api-access-257rm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901624 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901668 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed868f9d-f92d-4189-a052-22bde008e5f9-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901677 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901685 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901692 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed868f9d-f92d-4189-a052-22bde008e5f9-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901700 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:04.901843 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:04.901708 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed868f9d-f92d-4189-a052-22bde008e5f9-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:05.152128 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.152096 5894 generic.go:356] "Generic (PLEG): container finished" podID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerID="493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78" exitCode=0 Oct 30 07:58:05.152327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.152157 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" event={"ID":"ed868f9d-f92d-4189-a052-22bde008e5f9","Type":"ContainerDied","Data":"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78"} Oct 30 07:58:05.152327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.152224 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" event={"ID":"ed868f9d-f92d-4189-a052-22bde008e5f9","Type":"ContainerDied","Data":"0998b916f5b7b43360777333f993ca924a5e94d8b56244aff57d9d792cdd5bc4"} Oct 30 07:58:05.152327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.152159 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8" Oct 30 07:58:05.152327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.152248 5894 scope.go:122] "RemoveContainer" containerID="493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78" Oct 30 07:58:05.164669 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.164617 5894 scope.go:122] "RemoveContainer" containerID="493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78" Oct 30 07:58:05.164969 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:58:05.164945 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78\": container with ID starting with 493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78 not found: ID does not exist" containerID="493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78" Oct 30 07:58:05.165055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.164977 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78"} err="failed to get container status \"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78\": rpc error: code = NotFound desc = could not find container \"493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78\": container with ID starting with 493de4b25fa22609bdf6a34b883de0f59cd1896b05d66a87b6577da0199c8d78 not found: ID does not exist" Oct 30 07:58:05.213460 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.213426 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:58:05.225364 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:05.225330 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-759ae6-openshift-default-f56fd8fc9-52wh8"] Oct 30 07:58:06.234064 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:06.234026 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:06.432768 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:06.432734 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" path="/var/lib/kubelet/pods/ed868f9d-f92d-4189-a052-22bde008e5f9/volumes" Oct 30 07:58:07.841779 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:07.841743 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:09.278889 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:09.278856 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:09.513818 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:09.513783 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:11.122013 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:11.121977 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:11.249322 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:11.249280 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:12.219056 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:12.219008 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:12.861552 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:12.861521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:14.431513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:14.431480 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:15.604510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:15.604470 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:16.212436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:16.212401 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:17.843055 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:17.843020 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:18.384191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:18.384154 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:19.495431 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:19.495391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:21.037140 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:21.037103 5894 scope.go:122] "RemoveContainer" containerID="15f6d152f0f63c83d923330d254c9fb3170220245a4690b95adeb755ba2c5e9e" Oct 30 07:58:21.047923 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:21.047900 5894 scope.go:122] "RemoveContainer" containerID="e2477cceb712d69e92563f7990312bcfc94e8fadc1fe7b81c3ac59b2381561f0" Oct 30 07:58:21.177892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:21.177857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:22.835891 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:22.835858 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:23.373611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.373567 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9"] Oct 30 07:58:23.374696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374678 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" containerName="revoke-keys" Oct 30 07:58:23.374762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374700 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" containerName="revoke-keys" Oct 30 07:58:23.374762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374729 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" Oct 30 07:58:23.374762 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374739 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" Oct 30 07:58:23.374865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374784 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" Oct 30 07:58:23.374865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374792 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" Oct 30 07:58:23.374865 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374859 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" Oct 30 07:58:23.374968 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.374867 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" Oct 30 07:58:23.375031 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.375020 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed868f9d-f92d-4189-a052-22bde008e5f9" containerName="istio-proxy" Oct 30 07:58:23.375068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.375036 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22e9b00-cfd8-450d-bcc5-ec573a14f9ac" containerName="revoke-keys" Oct 30 07:58:23.375068 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.375052 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b224b8a5-9cf6-4cc1-a379-6dcc1e565cae" containerName="payload-processing" Oct 30 07:58:23.375139 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.375067 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcd7b22-7bca-4bb3-b72f-f714e13d02fb" containerName="payload-pre-processing" Oct 30 07:58:23.380625 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.380603 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.395326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.395298 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9"] Oct 30 07:58:23.403746 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.403697 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:58:23.403885 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.403763 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:58:23.469297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.469255 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.469503 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.469404 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dz7b\" (UniqueName: \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.469594 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.469571 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.570892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.570857 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dz7b\" (UniqueName: \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.571070 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.570966 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.571120 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.571089 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.571687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.571660 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.573543 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.573513 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.595422 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.595387 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dz7b\" (UniqueName: \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\") pod \"maas-api-revoke-sub-2c205912-qfhq9\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.698676 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.698555 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:23.931279 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.931246 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb"] Oct 30 07:58:23.937230 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.937202 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:23.956402 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.956324 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb"] Oct 30 07:58:23.977523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.977482 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:23.977776 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.977754 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snh9p\" (UniqueName: \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:23.977863 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:23.977809 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.016427 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:58:24.016397 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2734439e_f511_42e3_848a_bdc8101e2563.slice/crio-23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4 WatchSource:0}: Error finding container 23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4: Status 404 returned error can't find the container with id 23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4 Oct 30 07:58:24.017877 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.017852 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9"] Oct 30 07:58:24.080088 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.080060 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.080235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.080196 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snh9p\" (UniqueName: \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.080235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.080228 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.080774 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.080738 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.082520 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.082501 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.103716 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.103686 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snh9p\" (UniqueName: \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\") pod \"maas-api-revoke-sub-ada2ab1d-xjmbb\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.258884 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.258803 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:24.306288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.306244 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" event={"ID":"2734439e-f511-42e3-848a-bdc8101e2563","Type":"ContainerStarted","Data":"23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4"} Oct 30 07:58:24.530945 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.530905 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:24.606168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:24.606123 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb"] Oct 30 07:58:25.315862 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:25.315824 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" event={"ID":"c73c5261-5f2c-4b12-a2e1-50892e03ad54","Type":"ContainerStarted","Data":"d3126acceb77794669ac0ec36fc5d7d607494d2d6106d98d77411711b77b1c7f"} Oct 30 07:58:25.316337 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:25.315893 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" event={"ID":"c73c5261-5f2c-4b12-a2e1-50892e03ad54","Type":"ContainerStarted","Data":"c87e315e1c2bcd495743c20fbe289a318df5374fb4497e67b8b2f782ffcfa9ce"} Oct 30 07:58:25.317667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:25.317629 5894 generic.go:356] "Generic (PLEG): container finished" podID="2734439e-f511-42e3-848a-bdc8101e2563" containerID="acb5a6ae6c211cf6e4b3154c5e0ed931ca6e9dea424cec3b553498d7309326e6" exitCode=0 Oct 30 07:58:25.317791 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:25.317688 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" event={"ID":"2734439e-f511-42e3-848a-bdc8101e2563","Type":"ContainerDied","Data":"acb5a6ae6c211cf6e4b3154c5e0ed931ca6e9dea424cec3b553498d7309326e6"} Oct 30 07:58:25.357810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:25.357758 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" podStartSLOduration=2.357743456 podStartE2EDuration="2.357743456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:25.356777966 +0000 UTC m=+3669.047695404" watchObservedRunningTime="2026-10-30 07:58:25.357743456 +0000 UTC m=+3669.048660893" Oct 30 07:58:26.260373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.260338 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:26.352221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.352177 5894 generic.go:356] "Generic (PLEG): container finished" podID="c73c5261-5f2c-4b12-a2e1-50892e03ad54" containerID="d3126acceb77794669ac0ec36fc5d7d607494d2d6106d98d77411711b77b1c7f" exitCode=0 Oct 30 07:58:26.353260 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.352878 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" event={"ID":"c73c5261-5f2c-4b12-a2e1-50892e03ad54","Type":"ContainerDied","Data":"d3126acceb77794669ac0ec36fc5d7d607494d2d6106d98d77411711b77b1c7f"} Oct 30 07:58:26.535289 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.535262 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:26.620718 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.620684 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\") pod \"2734439e-f511-42e3-848a-bdc8101e2563\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " Oct 30 07:58:26.620934 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.620772 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\" (UniqueName: \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\") pod \"2734439e-f511-42e3-848a-bdc8101e2563\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " Oct 30 07:58:26.621022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.620996 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\") pod \"2734439e-f511-42e3-848a-bdc8101e2563\" (UID: \"2734439e-f511-42e3-848a-bdc8101e2563\") " Oct 30 07:58:26.621142 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.621121 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca" pod "2734439e-f511-42e3-848a-bdc8101e2563" (UID: "2734439e-f511-42e3-848a-bdc8101e2563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:26.621790 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.621763 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:26.623168 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.623142 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca" pod "2734439e-f511-42e3-848a-bdc8101e2563" (UID: "2734439e-f511-42e3-848a-bdc8101e2563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:26.623284 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.623261 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b" pod "2734439e-f511-42e3-848a-bdc8101e2563" (UID: "2734439e-f511-42e3-848a-bdc8101e2563"). InnerVolumeSpecName "kube-api-access-9dz7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:26.722786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.722751 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dz7b\" (UniqueName: \"kubernetes.io/projected/2734439e-f511-42e3-848a-bdc8101e2563-kube-api-access-9dz7b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:26.722786 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:26.722779 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2734439e-f511-42e3-848a-bdc8101e2563-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:27.361513 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.361482 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" Oct 30 07:58:27.362011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.361513 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9" event={"ID":"2734439e-f511-42e3-848a-bdc8101e2563","Type":"ContainerDied","Data":"23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4"} Oct 30 07:58:27.362011 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.361564 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23ded71b174bf402dfcb96a5e3aca19ad2e9382ce46988b8cc53256d750b35d4" Oct 30 07:58:27.528187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.528156 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:27.637766 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.637675 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\") pod \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " Oct 30 07:58:27.637957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.637813 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\" (UniqueName: \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\") pod \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " Oct 30 07:58:27.637957 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.637891 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\") pod \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\" (UID: \"c73c5261-5f2c-4b12-a2e1-50892e03ad54\") " Oct 30 07:58:27.638116 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.638087 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca" pod "c73c5261-5f2c-4b12-a2e1-50892e03ad54" (UID: "c73c5261-5f2c-4b12-a2e1-50892e03ad54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:27.638349 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.638332 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:27.640051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.640028 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca" pod "c73c5261-5f2c-4b12-a2e1-50892e03ad54" (UID: "c73c5261-5f2c-4b12-a2e1-50892e03ad54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:27.640131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.640048 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p" pod "c73c5261-5f2c-4b12-a2e1-50892e03ad54" (UID: "c73c5261-5f2c-4b12-a2e1-50892e03ad54"). InnerVolumeSpecName "kube-api-access-snh9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:27.740091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.740048 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c73c5261-5f2c-4b12-a2e1-50892e03ad54-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:27.740091 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.740083 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snh9p\" (UniqueName: \"kubernetes.io/projected/c73c5261-5f2c-4b12-a2e1-50892e03ad54-kube-api-access-snh9p\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:27.943307 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:27.943182 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:28.371147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:28.371108 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" Oct 30 07:58:28.371147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:28.371141 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb" event={"ID":"c73c5261-5f2c-4b12-a2e1-50892e03ad54","Type":"ContainerDied","Data":"c87e315e1c2bcd495743c20fbe289a318df5374fb4497e67b8b2f782ffcfa9ce"} Oct 30 07:58:28.371614 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:28.371194 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c87e315e1c2bcd495743c20fbe289a318df5374fb4497e67b8b2f782ffcfa9ce" Oct 30 07:58:29.555445 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:29.555409 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:31.230297 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:31.230254 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:32.095971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:32.095935 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:58:32.742739 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:32.742694 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:32.862784 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:32.862742 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:34.328005 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.327967 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:34.466612 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.466578 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:34.832170 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:58:34.832129 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:58:34.920570 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.920535 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz"] Oct 30 07:58:34.921499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921476 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c73c5261-5f2c-4b12-a2e1-50892e03ad54" containerName="revoke-keys" Oct 30 07:58:34.921499 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921499 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73c5261-5f2c-4b12-a2e1-50892e03ad54" containerName="revoke-keys" Oct 30 07:58:34.921799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921515 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2734439e-f511-42e3-848a-bdc8101e2563" containerName="revoke-keys" Oct 30 07:58:34.921799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921521 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2734439e-f511-42e3-848a-bdc8101e2563" containerName="revoke-keys" Oct 30 07:58:34.921799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921752 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73c5261-5f2c-4b12-a2e1-50892e03ad54" containerName="revoke-keys" Oct 30 07:58:34.921799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.921770 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2734439e-f511-42e3-848a-bdc8101e2563" containerName="revoke-keys" Oct 30 07:58:34.925906 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.925883 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:34.959052 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.959013 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:34.959244 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.959128 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbtj\" (UniqueName: \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:34.959325 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.959254 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:34.961896 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.961868 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz"] Oct 30 07:58:34.968871 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.968841 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:58:34.968997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:34.968975 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:58:35.060531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.060492 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.060754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.060547 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbtj\" (UniqueName: \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.060754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.060618 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.061357 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.061337 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.063257 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.063232 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.088664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.088574 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbtj\" (UniqueName: \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\") pod \"maas-api-revoke-sub-c32033fe-vdsqz\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.241314 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.241278 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:35.666584 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:58:35.666551 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c7cc6e_26d2_42f6_8b54_489b6f0693c4.slice/crio-333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b WatchSource:0}: Error finding container 333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b: Status 404 returned error can't find the container with id 333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b Oct 30 07:58:35.671472 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.671445 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz"] Oct 30 07:58:35.971200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:35.971123 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:36.129666 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.129594 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:36.276664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.276561 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:36.459480 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.459443 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" event={"ID":"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4","Type":"ContainerStarted","Data":"f55c7ca23390bba35c5fb18db607eaeff34f0e0d03444b316712665020982673"} Oct 30 07:58:36.459720 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.459505 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" event={"ID":"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4","Type":"ContainerStarted","Data":"333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b"} Oct 30 07:58:36.459878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.459848 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:36.510455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:36.510224 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" podStartSLOduration=2.510205215 podStartE2EDuration="2.510205215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:36.50810635 +0000 UTC m=+3680.199023790" watchObservedRunningTime="2026-10-30 07:58:36.510205215 +0000 UTC m=+3680.201122653" Oct 30 07:58:37.155705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:37.155666 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:37.301940 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:37.301889 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:37.468971 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:37.468885 5894 generic.go:356] "Generic (PLEG): container finished" podID="a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" containerID="f55c7ca23390bba35c5fb18db607eaeff34f0e0d03444b316712665020982673" exitCode=0 Oct 30 07:58:37.469154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:37.468970 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" event={"ID":"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4","Type":"ContainerDied","Data":"f55c7ca23390bba35c5fb18db607eaeff34f0e0d03444b316712665020982673"} Oct 30 07:58:38.249571 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.249482 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:38.654479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.654450 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:38.712125 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.712081 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\" (UniqueName: \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\") pod \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " Oct 30 07:58:38.712324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.712290 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\") pod \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " Oct 30 07:58:38.712403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.712353 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\") pod \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\" (UID: \"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4\") " Oct 30 07:58:38.713240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.712868 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca" pod "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" (UID: "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:38.713240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.713207 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:38.714963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.714885 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj" pod "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" (UID: "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4"). InnerVolumeSpecName "kube-api-access-8kbtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:38.718041 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.718017 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca" pod "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" (UID: "a7c7cc6e-26d2-42f6-8b54-489b6f0693c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:38.815057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.814964 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbtj\" (UniqueName: \"kubernetes.io/projected/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-kube-api-access-8kbtj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:38.815057 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:38.814999 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:39.030977 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.030942 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr"] Oct 30 07:58:39.034354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.034320 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec867198-f8qdr"] Oct 30 07:58:39.175316 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.175239 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl"] Oct 30 07:58:39.176086 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.176069 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" containerName="revoke-keys" Oct 30 07:58:39.176156 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.176086 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" containerName="revoke-keys" Oct 30 07:58:39.176195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.176178 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" containerName="revoke-keys" Oct 30 07:58:39.180436 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.180417 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.199044 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.199015 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl"] Oct 30 07:58:39.221182 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.221146 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.221367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.221210 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kwm6\" (UniqueName: \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.221418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.221371 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.322628 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.322588 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.323180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.322692 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kwm6\" (UniqueName: \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.323180 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.322763 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.323506 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.323484 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.325191 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.325168 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.346254 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.346226 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kwm6\" (UniqueName: \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\") pod \"maas-api-revoke-sub-9b65133b-m76xl\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.485457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.485379 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" Oct 30 07:58:39.485457 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.485414 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz" event={"ID":"a7c7cc6e-26d2-42f6-8b54-489b6f0693c4","Type":"ContainerDied","Data":"333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b"} Oct 30 07:58:39.485687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.485466 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="333714dad2d53bce12c52e99e149385837ac8711f8e043fb8156916ebd215b9b" Oct 30 07:58:39.499082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.499059 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:39.804671 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:58:39.804618 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa85c94e_1f58_4505_bd3a_d6d31c3ecb67.slice/crio-b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df WatchSource:0}: Error finding container b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df: Status 404 returned error can't find the container with id b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df Oct 30 07:58:39.806092 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.806067 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl"] Oct 30 07:58:39.959291 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:39.959251 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:40.433531 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:40.433451 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c65fd04-901d-4f6b-88da-e4768d6801b9" path="/var/lib/kubelet/pods/7c65fd04-901d-4f6b-88da-e4768d6801b9/volumes" Oct 30 07:58:40.494495 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:40.494458 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" event={"ID":"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67","Type":"ContainerStarted","Data":"2eb731023d2b7fb5de7484c1d254e56b992814a6b8632f511d5b41cadc389695"} Oct 30 07:58:40.494687 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:40.494518 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" event={"ID":"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67","Type":"ContainerStarted","Data":"b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df"} Oct 30 07:58:40.571483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:40.571429 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" podStartSLOduration=1.57141676 podStartE2EDuration="1.57141676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:58:40.567010916 +0000 UTC m=+3684.257928354" watchObservedRunningTime="2026-10-30 07:58:40.57141676 +0000 UTC m=+3684.262334198" Oct 30 07:58:41.504367 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:41.504336 5894 generic.go:356] "Generic (PLEG): container finished" podID="fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" containerID="2eb731023d2b7fb5de7484c1d254e56b992814a6b8632f511d5b41cadc389695" exitCode=0 Oct 30 07:58:41.504797 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:41.504423 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" event={"ID":"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67","Type":"ContainerDied","Data":"2eb731023d2b7fb5de7484c1d254e56b992814a6b8632f511d5b41cadc389695"} Oct 30 07:58:41.646569 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:41.646526 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:42.687764 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.687733 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:42.769606 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.769521 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\" (UniqueName: \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\") pod \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " Oct 30 07:58:42.769828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.769655 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\") pod \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " Oct 30 07:58:42.769828 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.769794 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\") pod \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\" (UID: \"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67\") " Oct 30 07:58:42.770082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.770048 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca" pod "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" (UID: "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:42.770487 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.770467 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:42.771925 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.771896 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca" pod "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" (UID: "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:42.772228 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.772211 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6" pod "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" (UID: "fa85c94e-1f58-4505-bd3a-d6d31c3ecb67"). InnerVolumeSpecName "kube-api-access-2kwm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:42.872324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.872286 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:42.872324 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:42.872317 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kwm6\" (UniqueName: \"kubernetes.io/projected/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67-kube-api-access-2kwm6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:43.436196 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:43.436158 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:43.527804 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:43.527764 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" event={"ID":"fa85c94e-1f58-4505-bd3a-d6d31c3ecb67","Type":"ContainerDied","Data":"b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df"} Oct 30 07:58:43.528022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:43.527823 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b93032de49e4cbb8d0d5d04c8b5c6a4695e850817bc96e6f322f6809b65fa4df" Oct 30 07:58:43.528022 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:43.527884 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl" Oct 30 07:58:44.228710 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.228644 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:58:44.420365 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.420321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:44.788879 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.788843 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd"] Oct 30 07:58:44.789795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.789774 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" containerName="revoke-keys" Oct 30 07:58:44.789795 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.789795 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" containerName="revoke-keys" Oct 30 07:58:44.789946 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.789937 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" containerName="revoke-keys" Oct 30 07:58:44.794317 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.794292 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:44.810428 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.810391 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:58:44.814618 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.814587 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd"] Oct 30 07:58:44.821714 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.821585 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:58:44.910584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.910548 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtcf\" (UniqueName: \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:44.910833 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.910812 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:44.910965 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:44.910947 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.012533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.012491 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.012744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.012560 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtcf\" (UniqueName: \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.012744 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.012721 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.013387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.013363 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.015082 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.015054 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.036589 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.036551 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtcf\" (UniqueName: \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\") pod \"maas-api-revoke-sub-4bc1de02-p74zd\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.111406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.111331 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:45.383189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.383100 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:45.449006 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.447305 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd"] Oct 30 07:58:45.548741 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:45.548704 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" event={"ID":"306f0d60-f2c5-49d3-9cea-8ea663a6857a","Type":"ContainerStarted","Data":"3cf4774cf39203db91aa4e6ffa20e42cf93cfc8d1681e0f916b4e0ca61b050d7"} Oct 30 07:58:46.560541 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:46.560504 5894 generic.go:356] "Generic (PLEG): container finished" podID="306f0d60-f2c5-49d3-9cea-8ea663a6857a" containerID="b069e44b2908871d7b2c74bf20e5d96e82a37e4be244831d4158e95dbab9ccc2" exitCode=0 Oct 30 07:58:46.560963 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:46.560589 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" event={"ID":"306f0d60-f2c5-49d3-9cea-8ea663a6857a","Type":"ContainerDied","Data":"b069e44b2908871d7b2c74bf20e5d96e82a37e4be244831d4158e95dbab9ccc2"} Oct 30 07:58:47.055033 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.054993 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:47.744339 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.744309 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:47.870974 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.870892 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\" (UniqueName: \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\") pod \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " Oct 30 07:58:47.871123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.871030 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\") pod \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " Oct 30 07:58:47.871123 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.871089 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\") pod \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\" (UID: \"306f0d60-f2c5-49d3-9cea-8ea663a6857a\") " Oct 30 07:58:47.871523 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.871486 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca" pod "306f0d60-f2c5-49d3-9cea-8ea663a6857a" (UID: "306f0d60-f2c5-49d3-9cea-8ea663a6857a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:47.873298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.873268 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca" pod "306f0d60-f2c5-49d3-9cea-8ea663a6857a" (UID: "306f0d60-f2c5-49d3-9cea-8ea663a6857a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:47.873298 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.873284 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf" pod "306f0d60-f2c5-49d3-9cea-8ea663a6857a" (UID: "306f0d60-f2c5-49d3-9cea-8ea663a6857a"). InnerVolumeSpecName "kube-api-access-4rtcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:47.973354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.973310 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtcf\" (UniqueName: \"kubernetes.io/projected/306f0d60-f2c5-49d3-9cea-8ea663a6857a-kube-api-access-4rtcf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:47.973354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.973344 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:47.973354 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:47.973359 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/306f0d60-f2c5-49d3-9cea-8ea663a6857a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:48.579467 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:48.579437 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" Oct 30 07:58:48.579682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:48.579432 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd" event={"ID":"306f0d60-f2c5-49d3-9cea-8ea663a6857a","Type":"ContainerDied","Data":"3cf4774cf39203db91aa4e6ffa20e42cf93cfc8d1681e0f916b4e0ca61b050d7"} Oct 30 07:58:48.579682 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:48.579582 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf4774cf39203db91aa4e6ffa20e42cf93cfc8d1681e0f916b4e0ca61b050d7" Oct 30 07:58:48.794294 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:48.794257 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:48.914176 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:48.914144 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:49.590238 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:49.590198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:50.428944 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:50.428908 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:52.113967 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.113928 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:52.589435 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.589396 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:52.858004 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.857922 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2"] Oct 30 07:58:52.858897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.858873 5894 cpu_manager.go:414] "containerMap: removing container" podUID="306f0d60-f2c5-49d3-9cea-8ea663a6857a" containerName="revoke-keys" Oct 30 07:58:52.858897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.858898 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="306f0d60-f2c5-49d3-9cea-8ea663a6857a" containerName="revoke-keys" Oct 30 07:58:52.859107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.859092 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="306f0d60-f2c5-49d3-9cea-8ea663a6857a" containerName="revoke-keys" Oct 30 07:58:52.867547 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.867524 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:52.886544 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.886514 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2"] Oct 30 07:58:52.902694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.900401 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:58:52.902694 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.901030 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:58:52.965240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.965190 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:52.965629 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.965568 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:52.965908 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:52.965796 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwml\" (UniqueName: \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.067143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.067101 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwml\" (UniqueName: \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.067385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.067277 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.067385 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.067357 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.068170 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.068137 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.070061 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.070036 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.091012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.090983 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwml\" (UniqueName: \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\") pod \"maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.197095 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.196997 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:53.518190 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:58:53.518152 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd305e36b_5a0c_47de_96ee_6d0e8cef3dfb.slice/crio-23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f WatchSource:0}: Error finding container 23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f: Status 404 returned error can't find the container with id 23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f Oct 30 07:58:53.519141 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.519112 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2"] Oct 30 07:58:53.627418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:53.627381 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" event={"ID":"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb","Type":"ContainerStarted","Data":"23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f"} Oct 30 07:58:54.060996 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:54.060911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:54.645808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:54.645243 5894 generic.go:356] "Generic (PLEG): container finished" podID="d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" containerID="17f4c68b8a55e63a9cd77819d2fff85866ff301768c01e4fda7be324a0dfe100" exitCode=0 Oct 30 07:58:54.645808 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:54.645314 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" event={"ID":"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb","Type":"ContainerDied","Data":"17f4c68b8a55e63a9cd77819d2fff85866ff301768c01e4fda7be324a0dfe100"} Oct 30 07:58:55.957705 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:55.953659 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:56.021657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.020762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\") pod \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " Oct 30 07:58:56.021657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.020847 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\") pod \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " Oct 30 07:58:56.021657 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.021067 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\" (UniqueName: \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\") pod \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\" (UID: \"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb\") " Oct 30 07:58:56.047689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.022190 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca" pod "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" (UID: "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:58:56.047689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.033104 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca" pod "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" (UID: "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:56.098664 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.074823 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml" pod "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" (UID: "d305e36b-5a0c-47de-96ee-6d0e8cef3dfb"). InnerVolumeSpecName "kube-api-access-sxwml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:56.175873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.162549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:56.176243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.176134 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxwml\" (UniqueName: \"kubernetes.io/projected/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-kube-api-access-sxwml\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:56.176243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.176195 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:56.176243 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.176211 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:56.651348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.651311 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:56.665659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.665606 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" Oct 30 07:58:56.665659 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.665652 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2" event={"ID":"d305e36b-5a0c-47de-96ee-6d0e8cef3dfb","Type":"ContainerDied","Data":"23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f"} Oct 30 07:58:56.665878 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:56.665699 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a24717d64084c03caacf51e9e81dc143bf6a3f84a9c39a2948392ca97b190f" Oct 30 07:58:58.089793 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:58.089751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:59.048708 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.048606 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656"] Oct 30 07:58:59.054440 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.054408 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-113c4f1a-pv656"] Oct 30 07:58:59.060721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.060691 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:58:59.102222 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.102186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:58:59.301163 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.301055 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" podUID="4f6e4b0e-cbeb-40b1-95cb-039608321116" containerName="main" containerID="cri-o://bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e" gracePeriod=45 Oct 30 07:58:59.398684 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.398575 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s"] Oct 30 07:58:59.399900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.399719 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" containerName="revoke-keys" Oct 30 07:58:59.399900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.399744 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" containerName="revoke-keys" Oct 30 07:58:59.400100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.400001 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" containerName="revoke-keys" Oct 30 07:58:59.406012 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.405990 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.422286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.420987 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:58:59.422286 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.421979 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s"] Oct 30 07:58:59.429045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.429017 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:58:59.507313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.506764 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.507313 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.506853 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnntn\" (UniqueName: \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.507611 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.507453 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.609899 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.609866 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.610072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.609973 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.610072 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.610006 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnntn\" (UniqueName: \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.610580 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.610558 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.612373 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.612350 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.634450 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.634417 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnntn\" (UniqueName: \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\") pod \"maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.640626 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.640591 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:58:59.700174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.696353 5894 generic.go:356] "Generic (PLEG): container finished" podID="4f6e4b0e-cbeb-40b1-95cb-039608321116" containerID="bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e" exitCode=0 Oct 30 07:58:59.700174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.696660 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" event={"ID":"4f6e4b0e-cbeb-40b1-95cb-039608321116","Type":"ContainerDied","Data":"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e"} Oct 30 07:58:59.700174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.696702 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" event={"ID":"4f6e4b0e-cbeb-40b1-95cb-039608321116","Type":"ContainerDied","Data":"b81bc21df1480e85f8290c3ee3eca4efc8a8c69860b7d310a6ba4a4a7726a11f"} Oct 30 07:58:59.700174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.696722 5894 scope.go:122] "RemoveContainer" containerID="bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e" Oct 30 07:58:59.700174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.696889 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf" Oct 30 07:58:59.713595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.713553 5894 scope.go:122] "RemoveContainer" containerID="bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e" Oct 30 07:58:59.713898 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.713869 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\" (UniqueName: \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.714001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.713942 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.714096 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:58:59.714062 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e\": container with ID starting with bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e not found: ID does not exist" containerID="bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e" Oct 30 07:58:59.714158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.714109 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e"} err="failed to get container status \"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e\": rpc error: code = NotFound desc = could not find container \"bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e\": container with ID starting with bedf99d05fd71ded76c8b1d6b677b74e043653d939b061612c8637b703fab37e not found: ID does not exist" Oct 30 07:58:59.714235 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.714198 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.714296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.714237 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.714296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.714272 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.714406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.714325 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\") pod \"4f6e4b0e-cbeb-40b1-95cb-039608321116\" (UID: \"4f6e4b0e-cbeb-40b1-95cb-039608321116\") " Oct 30 07:58:59.715233 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.715206 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:59.715418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.715400 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:59.715950 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.715922 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:59.718100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.718054 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:58:59.718100 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.718075 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:58:59.718954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.718929 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs" pod "4f6e4b0e-cbeb-40b1-95cb-039608321116" (UID: "4f6e4b0e-cbeb-40b1-95cb-039608321116"). InnerVolumeSpecName "kube-api-access-4bbjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:58:59.731131 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.731106 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821404 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821453 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f6e4b0e-cbeb-40b1-95cb-039608321116-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821470 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821488 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821507 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bbjs\" (UniqueName: \"kubernetes.io/projected/4f6e4b0e-cbeb-40b1-95cb-039608321116-kube-api-access-4bbjs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.821526 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f6e4b0e-cbeb-40b1-95cb-039608321116-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:58:59.943129 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:58:59.943032 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:00.057421 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:59:00.057390 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327215d8_fcfc_4f7b_b214_cd6d6ef84e40.slice/crio-9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e WatchSource:0}: Error finding container 9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e: Status 404 returned error can't find the container with id 9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e Oct 30 07:59:00.058188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.058162 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s"] Oct 30 07:59:00.061979 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.061952 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:59:00.066309 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.066285 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-057cfa/test-model-057cfa-kserve-5b786f8d6c-npfgf"] Oct 30 07:59:00.434800 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.434760 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6e4b0e-cbeb-40b1-95cb-039608321116" path="/var/lib/kubelet/pods/4f6e4b0e-cbeb-40b1-95cb-039608321116/volumes" Oct 30 07:59:00.435408 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.435385 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7800d6-4988-4410-b320-4b3871eefe57" path="/var/lib/kubelet/pods/6b7800d6-4988-4410-b320-4b3871eefe57/volumes" Oct 30 07:59:00.708649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.708534 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" event={"ID":"327215d8-fcfc-4f7b-b214-cd6d6ef84e40","Type":"ContainerStarted","Data":"c1f9bd115ea990fd0b68a8329fdc935fcfa1d118690ccbbd9b6ae9e4524240e3"} Oct 30 07:59:00.708649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.708619 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" event={"ID":"327215d8-fcfc-4f7b-b214-cd6d6ef84e40","Type":"ContainerStarted","Data":"9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e"} Oct 30 07:59:00.758378 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:00.758308 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" podStartSLOduration=1.758290624 podStartE2EDuration="1.758290624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:00.75187668 +0000 UTC m=+3704.442794118" watchObservedRunningTime="2026-10-30 07:59:00.758290624 +0000 UTC m=+3704.449208060" Oct 30 07:59:01.717410 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:01.717376 5894 generic.go:356] "Generic (PLEG): container finished" podID="327215d8-fcfc-4f7b-b214-cd6d6ef84e40" containerID="c1f9bd115ea990fd0b68a8329fdc935fcfa1d118690ccbbd9b6ae9e4524240e3" exitCode=0 Oct 30 07:59:01.717847 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:01.717515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" event={"ID":"327215d8-fcfc-4f7b-b214-cd6d6ef84e40","Type":"ContainerDied","Data":"c1f9bd115ea990fd0b68a8329fdc935fcfa1d118690ccbbd9b6ae9e4524240e3"} Oct 30 07:59:01.838288 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:01.838250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:02.897172 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.897143 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:59:02.969021 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.968991 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\") pod \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " Oct 30 07:59:02.969253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.969029 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\") pod \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " Oct 30 07:59:02.969253 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.969184 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\" (UniqueName: \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\") pod \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\" (UID: \"327215d8-fcfc-4f7b-b214-cd6d6ef84e40\") " Oct 30 07:59:02.969441 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.969418 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca" pod "327215d8-fcfc-4f7b-b214-cd6d6ef84e40" (UID: "327215d8-fcfc-4f7b-b214-cd6d6ef84e40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:02.969880 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.969851 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:02.971326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.971296 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca" pod "327215d8-fcfc-4f7b-b214-cd6d6ef84e40" (UID: "327215d8-fcfc-4f7b-b214-cd6d6ef84e40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:02.971424 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:02.971366 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn" pod "327215d8-fcfc-4f7b-b214-cd6d6ef84e40" (UID: "327215d8-fcfc-4f7b-b214-cd6d6ef84e40"). InnerVolumeSpecName "kube-api-access-jnntn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:03.071049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.071014 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:03.071049 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.071046 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnntn\" (UniqueName: \"kubernetes.io/projected/327215d8-fcfc-4f7b-b214-cd6d6ef84e40-kube-api-access-jnntn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:03.378897 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.378860 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:59:03.412696 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.412658 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:59:03.555893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.555861 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:03.685434 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.685340 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:03.734249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.734222 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" Oct 30 07:59:03.734403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.734256 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s" event={"ID":"327215d8-fcfc-4f7b-b214-cd6d6ef84e40","Type":"ContainerDied","Data":"9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e"} Oct 30 07:59:03.734403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:03.734303 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec7f0afa835542fc06c293b8e392c4a6fd0cc5b81eaf063ea79ced7e726252e" Oct 30 07:59:04.829170 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:04.829131 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:59:05.207591 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:05.207497 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:06.914418 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:06.914376 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:08.379738 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:08.379671 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" containerID="cri-o://bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf" gracePeriod=25 Oct 30 07:59:08.412941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:08.412893 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" containerID="cri-o://0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad" gracePeriod=25 Oct 30 07:59:08.545251 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:08.545221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:09.939310 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:09.939278 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:59:09.980483 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:09.980440 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:59:10.107854 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:10.107821 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:10.231117 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:10.231016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:10.351319 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:10.351280 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:11.958461 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:11.958425 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:13.668992 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:13.668956 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:14.086830 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.086784 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" podUID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" containerName="main" containerID="cri-o://a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa" gracePeriod=45 Oct 30 07:59:14.290702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.290588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:14.371067 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.371038 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:59:14.447533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447495 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.447755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447547 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.447755 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447677 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.447901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447798 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.447901 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447854 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\" (UniqueName: \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.448014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447942 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:14.448014 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.447954 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\") pod \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\" (UID: \"2438141b-a7d8-4ea7-b0fe-55788b0713b1\") " Oct 30 07:59:14.448165 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.448145 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:14.448241 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.448162 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:14.448900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.448870 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.449037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.448903 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.449037 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.448917 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.450275 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.450242 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:59:14.450832 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.450811 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "kube-api-access-7jcrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:14.450942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.450871 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs" pod "2438141b-a7d8-4ea7-b0fe-55788b0713b1" (UID: "2438141b-a7d8-4ea7-b0fe-55788b0713b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:14.550845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.550809 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2438141b-a7d8-4ea7-b0fe-55788b0713b1-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.550845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.550838 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2438141b-a7d8-4ea7-b0fe-55788b0713b1-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.550845 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.550849 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jcrb\" (UniqueName: \"kubernetes.io/projected/2438141b-a7d8-4ea7-b0fe-55788b0713b1-kube-api-access-7jcrb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:14.825107 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.825072 5894 generic.go:356] "Generic (PLEG): container finished" podID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" containerID="a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa" exitCode=0 Oct 30 07:59:14.825573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.825125 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" Oct 30 07:59:14.825573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.825145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" event={"ID":"2438141b-a7d8-4ea7-b0fe-55788b0713b1","Type":"ContainerDied","Data":"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa"} Oct 30 07:59:14.825573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.825191 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb" event={"ID":"2438141b-a7d8-4ea7-b0fe-55788b0713b1","Type":"ContainerDied","Data":"4adff0e9d44143535ab6eb7c555d9c9f9febaa5243ce73f00205ed7d8a771d57"} Oct 30 07:59:14.825573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.825208 5894 scope.go:122] "RemoveContainer" containerID="a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa" Oct 30 07:59:14.837158 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.837134 5894 scope.go:122] "RemoveContainer" containerID="a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa" Oct 30 07:59:14.837437 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:14.837415 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa\": container with ID starting with a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa not found: ID does not exist" containerID="a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa" Oct 30 07:59:14.837533 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.837444 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa"} err="failed to get container status \"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa\": rpc error: code = NotFound desc = could not find container \"a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa\": container with ID starting with a6e7ae6f5fd2044ce12a4036c1a96d0c1b9fe2804830b1ce19ffcc7c82571dfa not found: ID does not exist" Oct 30 07:59:14.879080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.879043 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:59:14.891500 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.891473 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-352fb3/test-model-352fb3-kserve-677bd8bfd9-lzrvb"] Oct 30 07:59:14.939568 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.939528 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" podUID="37550a32-f812-464b-8543-4631f9461985" containerName="payload-processing" containerID="cri-o://002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f" gracePeriod=25 Oct 30 07:59:14.973080 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.973044 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:14.981311 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:14.981261 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" containerID="cri-o://ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90" gracePeriod=25 Oct 30 07:59:15.234462 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.234433 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:59:15.362200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.362120 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\" (UniqueName: \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\") pod \"37550a32-f812-464b-8543-4631f9461985\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " Oct 30 07:59:15.362200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.362159 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\") pod \"37550a32-f812-464b-8543-4631f9461985\" (UID: \"37550a32-f812-464b-8543-4631f9461985\") " Oct 30 07:59:15.362693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.362621 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume" pod "37550a32-f812-464b-8543-4631f9461985" (UID: "37550a32-f812-464b-8543-4631f9461985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:15.364442 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.364412 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk" pod "37550a32-f812-464b-8543-4631f9461985" (UID: "37550a32-f812-464b-8543-4631f9461985"). InnerVolumeSpecName "kube-api-access-pv5wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:15.396401 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.396367 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:15.475143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.466538 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv5wk\" (UniqueName: \"kubernetes.io/projected/37550a32-f812-464b-8543-4631f9461985-kube-api-access-pv5wk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:15.475143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.466574 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37550a32-f812-464b-8543-4631f9461985-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:15.475143 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.473110 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 07:59:15.835651 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.835607 5894 generic.go:356] "Generic (PLEG): container finished" podID="37550a32-f812-464b-8543-4631f9461985" containerID="002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f" exitCode=0 Oct 30 07:59:15.836094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.835690 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" Oct 30 07:59:15.836094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.835771 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" event={"ID":"37550a32-f812-464b-8543-4631f9461985","Type":"ContainerDied","Data":"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f"} Oct 30 07:59:15.836094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.835827 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd" event={"ID":"37550a32-f812-464b-8543-4631f9461985","Type":"ContainerDied","Data":"0a5d5b31790ca9502cc91b5fb8b06060d3208d0d5a873bb4deab54154179d0f2"} Oct 30 07:59:15.836094 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.835852 5894 scope.go:122] "RemoveContainer" containerID="002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f" Oct 30 07:59:15.852076 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.851612 5894 scope.go:122] "RemoveContainer" containerID="002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f" Oct 30 07:59:15.852277 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:15.852254 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f\": container with ID starting with 002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f not found: ID does not exist" containerID="002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f" Oct 30 07:59:15.852348 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.852289 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f"} err="failed to get container status \"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f\": rpc error: code = NotFound desc = could not find container \"002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f\": container with ID starting with 002fc950d5fa41aa0498b4dfcef4d3c64d0878f38e69bf150c131418e32a718f not found: ID does not exist" Oct 30 07:59:15.895293 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.895259 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:59:15.908112 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.908071 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-057cfa-5794fbc596-b97fd"] Oct 30 07:59:15.970200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:15.970162 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:16.439051 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:16.439016 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" path="/var/lib/kubelet/pods/2438141b-a7d8-4ea7-b0fe-55788b0713b1/volumes" Oct 30 07:59:16.440195 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:16.439733 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37550a32-f812-464b-8543-4631f9461985" path="/var/lib/kubelet/pods/37550a32-f812-464b-8543-4631f9461985/volumes" Oct 30 07:59:17.465658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:17.458146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:19.354668 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:19.354600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:21.095479 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:21.095428 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:21.103546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:21.103522 5894 scope.go:122] "RemoveContainer" containerID="a7abaf2a7cc1cc57f4bcee4d46a7f227ff2ab7fd1734922441614e26996f1825" Oct 30 07:59:22.807406 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:22.807372 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:22.885942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:22.885903 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:24.408444 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:24.408404 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:24.961054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:24.961020 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:25.970269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:25.970230 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:26.054187 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:26.054146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:27.647864 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:27.647824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:28.445883 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:28.445843 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode923e6a7_aced_439e_81d2_df142d5f7c82.slice/crio-bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf.scope\": RecentStats: unable to find data in memory cache]" Oct 30 07:59:28.722525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.722496 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:59:28.781455 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.781419 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:59:28.787425 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.787393 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\") pod \"e923e6a7-aced-439e-81d2-df142d5f7c82\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " Oct 30 07:59:28.787649 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.787616 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\" (UniqueName: \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\") pod \"e923e6a7-aced-439e-81d2-df142d5f7c82\" (UID: \"e923e6a7-aced-439e-81d2-df142d5f7c82\") " Oct 30 07:59:28.787873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.787843 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume" pod "e923e6a7-aced-439e-81d2-df142d5f7c82" (UID: "e923e6a7-aced-439e-81d2-df142d5f7c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:28.788221 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.788200 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e923e6a7-aced-439e-81d2-df142d5f7c82-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:28.789760 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.789736 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg" pod "e923e6a7-aced-439e-81d2-df142d5f7c82" (UID: "e923e6a7-aced-439e-81d2-df142d5f7c82"). InnerVolumeSpecName "kube-api-access-nmvtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:28.889190 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.889154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\") pod \"5636aab6-f0d3-4afd-9029-7529e2c130b8\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " Oct 30 07:59:28.889387 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.889252 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\") pod \"5636aab6-f0d3-4afd-9029-7529e2c130b8\" (UID: \"5636aab6-f0d3-4afd-9029-7529e2c130b8\") " Oct 30 07:59:28.889546 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.889518 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume" pod "5636aab6-f0d3-4afd-9029-7529e2c130b8" (UID: "5636aab6-f0d3-4afd-9029-7529e2c130b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:28.889730 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.889715 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5636aab6-f0d3-4afd-9029-7529e2c130b8-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:28.889794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.889732 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmvtg\" (UniqueName: \"kubernetes.io/projected/e923e6a7-aced-439e-81d2-df142d5f7c82-kube-api-access-nmvtg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:28.891475 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.891448 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8" pod "5636aab6-f0d3-4afd-9029-7529e2c130b8" (UID: "5636aab6-f0d3-4afd-9029-7529e2c130b8"). InnerVolumeSpecName "kube-api-access-gfth8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:28.975941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.975854 5894 generic.go:356] "Generic (PLEG): container finished" podID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerID="0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad" exitCode=0 Oct 30 07:59:28.975941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.975901 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" Oct 30 07:59:28.976181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.975935 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" event={"ID":"5636aab6-f0d3-4afd-9029-7529e2c130b8","Type":"ContainerDied","Data":"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad"} Oct 30 07:59:28.976181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.975991 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4" event={"ID":"5636aab6-f0d3-4afd-9029-7529e2c130b8","Type":"ContainerDied","Data":"193102b35eb20046a0009869625b4dcbd630571141c557a348edae6ef85507c5"} Oct 30 07:59:28.976181 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.976049 5894 scope.go:122] "RemoveContainer" containerID="0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad" Oct 30 07:59:28.978001 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.977974 5894 generic.go:356] "Generic (PLEG): container finished" podID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerID="bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf" exitCode=0 Oct 30 07:59:28.978130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.978012 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" event={"ID":"e923e6a7-aced-439e-81d2-df142d5f7c82","Type":"ContainerDied","Data":"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf"} Oct 30 07:59:28.978130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.978031 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" Oct 30 07:59:28.978130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.978050 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk" event={"ID":"e923e6a7-aced-439e-81d2-df142d5f7c82","Type":"ContainerDied","Data":"dc624cb888b4ab7719b593bb893da873f57f255f74dcde259df21801f65e8d69"} Oct 30 07:59:28.988501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.988482 5894 scope.go:122] "RemoveContainer" containerID="0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad" Oct 30 07:59:28.988792 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:28.988771 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad\": container with ID starting with 0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad not found: ID does not exist" containerID="0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad" Oct 30 07:59:28.988886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.988803 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad"} err="failed to get container status \"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad\": rpc error: code = NotFound desc = could not find container \"0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad\": container with ID starting with 0719d6152aa1662ed0cd4d10901022c5338d61d342c1135e3658d8b48aaf4cad not found: ID does not exist" Oct 30 07:59:28.988886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.988826 5894 scope.go:122] "RemoveContainer" containerID="bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf" Oct 30 07:59:28.991497 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.991472 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfth8\" (UniqueName: \"kubernetes.io/projected/5636aab6-f0d3-4afd-9029-7529e2c130b8-kube-api-access-gfth8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:28.998721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.998704 5894 scope.go:122] "RemoveContainer" containerID="bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf" Oct 30 07:59:28.998983 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:28.998964 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf\": container with ID starting with bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf not found: ID does not exist" containerID="bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf" Oct 30 07:59:28.999047 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:28.998993 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf"} err="failed to get container status \"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf\": rpc error: code = NotFound desc = could not find container \"bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf\": container with ID starting with bceef0bdb640dc24c149177031e03e1f37b01abb5024366b89329ee1e4e4decf not found: ID does not exist" Oct 30 07:59:29.031130 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:29.031096 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:59:29.034586 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:29.034555 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51147f-8896f55bd-x4cbk"] Oct 30 07:59:29.069188 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:29.069156 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:59:29.072758 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:29.072728 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51147f-74c7c6568c-p6lv4"] Oct 30 07:59:29.396304 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:29.396266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:30.433040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:30.432995 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" path="/var/lib/kubelet/pods/5636aab6-f0d3-4afd-9029-7529e2c130b8/volumes" Oct 30 07:59:30.433801 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:30.433778 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" path="/var/lib/kubelet/pods/e923e6a7-aced-439e-81d2-df142d5f7c82/volumes" Oct 30 07:59:31.139648 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:31.139609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:32.728856 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:32.728819 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:32.870296 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:32.870252 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:32.885501 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:32.885468 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:59:33.181166 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:33.181115 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:34.437997 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:34.437960 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:34.830691 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:34.830623 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 07:59:35.272333 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.272303 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:59:35.380650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.380548 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\") pod \"40ef68d9-11ac-4808-849a-fb681a4b01a2\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " Oct 30 07:59:35.380650 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.380607 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\") pod \"40ef68d9-11ac-4808-849a-fb681a4b01a2\" (UID: \"40ef68d9-11ac-4808-849a-fb681a4b01a2\") " Oct 30 07:59:35.381042 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.381017 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume" pod "40ef68d9-11ac-4808-849a-fb681a4b01a2" (UID: "40ef68d9-11ac-4808-849a-fb681a4b01a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:35.382954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.382931 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7" pod "40ef68d9-11ac-4808-849a-fb681a4b01a2" (UID: "40ef68d9-11ac-4808-849a-fb681a4b01a2"). InnerVolumeSpecName "kube-api-access-vw5q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:35.482933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.482899 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/40ef68d9-11ac-4808-849a-fb681a4b01a2-kube-api-access-vw5q7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:35.482933 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:35.482927 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40ef68d9-11ac-4808-849a-fb681a4b01a2-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:36.033754 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.033715 5894 generic.go:356] "Generic (PLEG): container finished" podID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerID="ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90" exitCode=0 Oct 30 07:59:36.033959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.033765 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" Oct 30 07:59:36.033959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.033774 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" event={"ID":"40ef68d9-11ac-4808-849a-fb681a4b01a2","Type":"ContainerDied","Data":"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90"} Oct 30 07:59:36.033959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.033828 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv" event={"ID":"40ef68d9-11ac-4808-849a-fb681a4b01a2","Type":"ContainerDied","Data":"1c8a405c0d1e6b51395788f4bcfec0affa3741b59f5c71d672d0c5dc9382d48a"} Oct 30 07:59:36.033959 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.033856 5894 scope.go:122] "RemoveContainer" containerID="ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90" Oct 30 07:59:36.045496 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.045474 5894 scope.go:122] "RemoveContainer" containerID="ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90" Oct 30 07:59:36.045819 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:36.045794 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90\": container with ID starting with ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90 not found: ID does not exist" containerID="ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90" Oct 30 07:59:36.045868 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.045828 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90"} err="failed to get container status \"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90\": rpc error: code = NotFound desc = could not find container \"ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90\": container with ID starting with ee801e9df99642c757ece7424fa95e012f63679626c31023d4bb7d42958c4a90 not found: ID does not exist" Oct 30 07:59:36.095753 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.095720 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:59:36.107794 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.107762 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-057cfa-7fd88cf4ff-6rpcv"] Oct 30 07:59:36.140788 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.140756 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:36.432548 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:36.432460 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" path="/var/lib/kubelet/pods/40ef68d9-11ac-4808-849a-fb681a4b01a2/volumes" Oct 30 07:59:37.796096 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:37.796060 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:39.534430 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:39.534385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:39.676942 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:39.676909 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:40.245881 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:40.244857 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:40.468258 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:40.468213 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:41.354174 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:41.354137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:42.374200 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:42.374166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:42.523567 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:42.523523 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:42.721582 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:42.721493 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:43.160840 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:43.160797 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:43.832287 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:43.832243 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:43.955883 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:43.955849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:45.037525 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:45.037485 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:45.698953 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:45.698898 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:45.826236 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:45.826190 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:46.848438 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:46.848403 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:48.308938 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:48.308887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:48.902154 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:48.902099 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:49.019573 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.019533 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:49.050647 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.050587 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s"] Oct 30 07:59:49.062769 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.062730 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-057cfa-n2w5s"] Oct 30 07:59:49.131025 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.130996 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:49.567693 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.567653 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:49.683326 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:49.683289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:50.433327 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:50.433281 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327215d8-fcfc-4f7b-b214-cd6d6ef84e40" path="/var/lib/kubelet/pods/327215d8-fcfc-4f7b-b214-cd6d6ef84e40/volumes" Oct 30 07:59:50.662873 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:50.662832 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:59:50.663382 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:50.663092 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" podUID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" containerName="maas-api" containerID="cri-o://782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181" gracePeriod=30 Oct 30 07:59:50.754147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:50.754113 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:51.097951 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.097916 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:59:51.181721 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.181652 5894 generic.go:356] "Generic (PLEG): container finished" podID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" containerID="782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181" exitCode=0 Oct 30 07:59:51.181941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.181860 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" event={"ID":"d3ab756e-5453-465e-bbd3-055ffb3f10a4","Type":"ContainerDied","Data":"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181"} Oct 30 07:59:51.181941 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.181912 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" event={"ID":"d3ab756e-5453-465e-bbd3-055ffb3f10a4","Type":"ContainerDied","Data":"05c942b08000e37082a938aff7236d5cb18376a6693b85dd9ea22bd8145bcfa4"} Oct 30 07:59:51.182059 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.181942 5894 scope.go:122] "RemoveContainer" containerID="782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181" Oct 30 07:59:51.182145 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.182130 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb" Oct 30 07:59:51.191329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.191295 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\") pod \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " Oct 30 07:59:51.191584 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.191564 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\") pod \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " Oct 30 07:59:51.191702 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.191682 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\") pod \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " Oct 30 07:59:51.192138 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.192108 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca" pod "d3ab756e-5453-465e-bbd3-055ffb3f10a4" (UID: "d3ab756e-5453-465e-bbd3-055ffb3f10a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:59:51.192240 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.192164 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\" (UniqueName: \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\") pod \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\" (UID: \"d3ab756e-5453-465e-bbd3-055ffb3f10a4\") " Oct 30 07:59:51.192827 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.192801 5894 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3ab756e-5453-465e-bbd3-055ffb3f10a4-openshift-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:51.194717 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.194691 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs" pod "d3ab756e-5453-465e-bbd3-055ffb3f10a4" (UID: "d3ab756e-5453-465e-bbd3-055ffb3f10a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:51.194887 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.194871 5894 scope.go:122] "RemoveContainer" containerID="782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181" Oct 30 07:59:51.195281 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.195261 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls" pod "d3ab756e-5453-465e-bbd3-055ffb3f10a4" (UID: "d3ab756e-5453-465e-bbd3-055ffb3f10a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:59:51.195376 ip-10-0-143-90 kubenswrapper[5894]: E1030 07:59:51.195318 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181\": container with ID starting with 782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181 not found: ID does not exist" containerID="782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181" Oct 30 07:59:51.195376 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.195356 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181"} err="failed to get container status \"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181\": rpc error: code = NotFound desc = could not find container \"782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181\": container with ID starting with 782d97acc6cd15d849eeaf6607498313d3409d26df9acc07077ceff9d68cd181 not found: ID does not exist" Oct 30 07:59:51.195814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.195788 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b" pod "d3ab756e-5453-465e-bbd3-055ffb3f10a4" (UID: "d3ab756e-5453-465e-bbd3-055ffb3f10a4"). InnerVolumeSpecName "kube-api-access-zfk9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:59:51.310038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.309885 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-maas-api-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:51.310038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.309932 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfk9b\" (UniqueName: \"kubernetes.io/projected/d3ab756e-5453-465e-bbd3-055ffb3f10a4-kube-api-access-zfk9b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:51.310038 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.309948 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3ab756e-5453-465e-bbd3-055ffb3f10a4-metrics-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 07:59:51.399658 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.399098 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400257 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400286 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400327 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4f6e4b0e-cbeb-40b1-95cb-039608321116" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400336 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6e4b0e-cbeb-40b1-95cb-039608321116" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400367 5894 cpu_manager.go:414] "containerMap: removing container" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400376 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400390 5894 cpu_manager.go:414] "containerMap: removing container" podUID="327215d8-fcfc-4f7b-b214-cd6d6ef84e40" containerName="revoke-keys" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400398 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327215d8-fcfc-4f7b-b214-cd6d6ef84e40" containerName="revoke-keys" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400411 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" containerName="maas-api" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400419 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" containerName="maas-api" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400447 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400455 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400479 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400487 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400499 5894 cpu_manager.go:414] "containerMap: removing container" podUID="37550a32-f812-464b-8543-4631f9461985" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400506 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37550a32-f812-464b-8543-4631f9461985" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400703 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6e4b0e-cbeb-40b1-95cb-039608321116" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400729 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ef68d9-11ac-4808-849a-fb681a4b01a2" containerName="payload-pre-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400741 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="327215d8-fcfc-4f7b-b214-cd6d6ef84e40" containerName="revoke-keys" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400754 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" containerName="maas-api" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400766 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e923e6a7-aced-439e-81d2-df142d5f7c82" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400780 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2438141b-a7d8-4ea7-b0fe-55788b0713b1" containerName="main" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400790 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="37550a32-f812-464b-8543-4631f9461985" containerName="payload-processing" Oct 30 07:59:51.403792 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.400802 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5636aab6-f0d3-4afd-9029-7529e2c130b8" containerName="payload-pre-processing" Oct 30 07:59:51.408234 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.408203 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.419595 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.418376 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 07:59:51.441153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.429995 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"kube-root-ca.crt\"" Oct 30 07:59:51.441153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.430726 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"default-dockercfg-52dcn\"" Oct 30 07:59:51.441153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.430838 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"sub-test-model-bf563a-kserve-self-signed-certs\"" Oct 30 07:59:51.441153 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.430916 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bf563a\"/\"openshift-service-ca.crt\"" Oct 30 07:59:51.467948 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.467916 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:51.516492 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516445 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.516719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516571 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.516719 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516658 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-774k9\" (UniqueName: \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.516814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516768 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.516814 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516790 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.516954 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.516941 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.539893 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.539857 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:59:51.553459 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.553424 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-51147f-65f5b7664b-cjnrb"] Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.625102 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626177 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626221 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626349 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626443 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626505 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.626540 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-774k9\" (UniqueName: \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.627310 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.634249 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.627572 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.648507 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.647981 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.674984 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.674945 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-774k9\" (UniqueName: \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.676045 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.676016 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.687667 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.685231 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\") pod \"sub-test-model-bf563a-kserve-fbf5dd867-799qx\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.747069 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.747031 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:51.756147 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.756117 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:51.819512 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.819032 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2"] Oct 30 07:59:51.834655 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.831481 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51147f-ltqc2"] Oct 30 07:59:51.886505 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:51.886351 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:52.136863 ip-10-0-143-90 kubenswrapper[5894]: W1030 07:59:52.136787 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffaf9e24_8a22_4cf5_8595_40e092726864.slice/crio-f8694fbb4383922268ef22e8c7f4f2fd31a4e8c168329b1ff9a1c3c6097f6bd0 WatchSource:0}: Error finding container f8694fbb4383922268ef22e8c7f4f2fd31a4e8c168329b1ff9a1c3c6097f6bd0: Status 404 returned error can't find the container with id f8694fbb4383922268ef22e8c7f4f2fd31a4e8c168329b1ff9a1c3c6097f6bd0 Oct 30 07:59:52.137622 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:52.137595 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 07:59:52.194157 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:52.194126 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" event={"ID":"ffaf9e24-8a22-4cf5-8595-40e092726864","Type":"ContainerStarted","Data":"f8694fbb4383922268ef22e8c7f4f2fd31a4e8c168329b1ff9a1c3c6097f6bd0"} Oct 30 07:59:52.325911 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:52.325872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:52.434690 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:52.434592 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d305e36b-5a0c-47de-96ee-6d0e8cef3dfb" path="/var/lib/kubelet/pods/d305e36b-5a0c-47de-96ee-6d0e8cef3dfb/volumes" Oct 30 07:59:52.435189 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:52.435167 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ab756e-5453-465e-bbd3-055ffb3f10a4" path="/var/lib/kubelet/pods/d3ab756e-5453-465e-bbd3-055ffb3f10a4/volumes" Oct 30 07:59:53.207350 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:53.207293 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" event={"ID":"ffaf9e24-8a22-4cf5-8595-40e092726864","Type":"ContainerStarted","Data":"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101"} Oct 30 07:59:53.261780 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:53.261721 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" podStartSLOduration=2.261704711 podStartE2EDuration="2.261704711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:53.259758216 +0000 UTC m=+3756.950675654" watchObservedRunningTime="2026-10-30 07:59:53.261704711 +0000 UTC m=+3756.952622148" Oct 30 07:59:53.590890 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:53.590854 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:53.747219 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:53.747178 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 07:59:55.336886 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:55.336850 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:55.880743 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:55.880703 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:55.992789 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:55.992759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:56.273689 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:56.273571 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:56.911892 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:56.911855 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn"] Oct 30 07:59:56.917675 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:56.917651 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:56.935465 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:56.935440 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn"] Oct 30 07:59:57.016575 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.016544 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:59:57.016799 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.016781 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 07:59:57.018403 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.018376 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fztnp\" (UniqueName: \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.018508 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.018494 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.018654 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.018611 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.120155 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.120118 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.120329 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.120290 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fztnp\" (UniqueName: \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.120407 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.120391 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.121085 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.121057 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.122510 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.122486 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.144040 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.144017 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fztnp\" (UniqueName: \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\") pod \"maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.233514 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.233438 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 07:59:57.365269 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.365237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:57.559810 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:57.559587 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn"] Oct 30 07:59:58.254471 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:58.254382 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" event={"ID":"8c9ef108-46fa-458b-af70-90065ca8c203","Type":"ContainerStarted","Data":"aedbad0c82d23573ecfdbb12a1d06c03563d21eca6fe4a41398309c7c10549f9"} Oct 30 07:59:58.254900 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:58.254474 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" event={"ID":"8c9ef108-46fa-458b-af70-90065ca8c203","Type":"ContainerStarted","Data":"fa80849176e0f30d757f5ddaf919b027bfb53b42b0af4c79b02c783544106478"} Oct 30 07:59:58.296872 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:58.296810 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" podStartSLOduration=2.296792005 podStartE2EDuration="2.296792005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:59:58.294911461 +0000 UTC m=+3761.985828898" watchObservedRunningTime="2026-10-30 07:59:58.296792005 +0000 UTC m=+3761.987709446" Oct 30 07:59:59.141978 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.141940 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:59.263447 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.263413 5894 generic.go:356] "Generic (PLEG): container finished" podID="8c9ef108-46fa-458b-af70-90065ca8c203" containerID="aedbad0c82d23573ecfdbb12a1d06c03563d21eca6fe4a41398309c7c10549f9" exitCode=0 Oct 30 07:59:59.263926 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.263504 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" event={"ID":"8c9ef108-46fa-458b-af70-90065ca8c203","Type":"ContainerDied","Data":"aedbad0c82d23573ecfdbb12a1d06c03563d21eca6fe4a41398309c7c10549f9"} Oct 30 07:59:59.265421 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.265392 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:59.383920 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.383773 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 07:59:59.683054 ip-10-0-143-90 kubenswrapper[5894]: I1030 07:59:59.683017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:00.163155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.163119 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:00.167831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.167803 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:00.168011 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.167900 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.172433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.172409 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:00.172579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.172525 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.177255 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.177226 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.177400 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.177285 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.177495 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.177415 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df24n\" (UniqueName: \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.177883 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.177852 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96hpd\" (UniqueName: \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.178077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.177899 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:00.178444 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.178163 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.182966 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.182947 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:00.183113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.183094 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.188450 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.188430 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.201862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.201526 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:00.228655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.220324 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:00.228655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.226486 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.234940 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.234905 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:00.238015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.237990 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:00.243758 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.243736 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:00.248480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.248457 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:00.255784 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.255347 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:00.261865 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.261275 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:00.261865 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.261394 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.267319 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.267293 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:00.267857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.267832 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.283809 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.279972 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.283917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.283798 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df24n\" (UniqueName: \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.283917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.283895 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.283942 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qw4c\" (UniqueName: \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284024 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96hpd\" (UniqueName: \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284085 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtcq\" (UniqueName: \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284132 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tl2c\" (UniqueName: \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284192 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284259 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lmq\" (UniqueName: \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284292 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284334 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284381 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284416 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284462 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284517 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284585 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nl7\" (UniqueName: \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.284654 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqtk\" (UniqueName: \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.286371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.286158 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.290368 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.290341 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.303803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.303765 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:00.312384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.312188 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:00.328127 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.327895 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:00.353004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.346077 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:00.353004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.351821 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:00.358177 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.357467 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96hpd\" (UniqueName: \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\") pod \"maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.358342 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.358297 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.362804 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.361777 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df24n\" (UniqueName: \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\") pod \"maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.376770 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.376722 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.385856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqtk\" (UniqueName: \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.385954 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.385998 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qw4c\" (UniqueName: \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386072 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtcq\" (UniqueName: \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386101 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386138 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tl2c\" (UniqueName: \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386189 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386246 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lmq\" (UniqueName: \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386271 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386306 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386344 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386379 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386433 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwgrk\" (UniqueName: \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386464 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476k4\" (UniqueName: \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386495 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.386697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.386542 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nl7\" (UniqueName: \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.387855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.387825 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.389149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.389116 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.389796 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.389768 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.389917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.389778 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.390121 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.390098 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.393620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.393593 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.393995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.393972 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:00.394687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.394126 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.402678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.402490 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:00.407267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.407238 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.408302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.408283 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:00.408387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.408327 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:00.408425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.408410 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.412041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.411979 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:00.424831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.423386 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:00.427091 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.426878 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqtk\" (UniqueName: \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\") pod \"maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.438702 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.437074 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qw4c\" (UniqueName: \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.440330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.440299 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nl7\" (UniqueName: \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\") pod \"maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.440937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.440907 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtcq\" (UniqueName: \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\") pod \"maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.441860 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.441829 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tl2c\" (UniqueName: \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\") pod \"maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.443676 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.443614 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lmq\" (UniqueName: \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\") pod \"maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.444205 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.444181 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg"] Oct 30 08:00:00.465014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.464614 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:00.465414 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.464650 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.472656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.472621 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.480660 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.479719 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg"] Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.488711 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.490689 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.491823 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4n66\" (UniqueName: \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.491974 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkxw\" (UniqueName: \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.492091 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwgrk\" (UniqueName: \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.492124 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476k4\" (UniqueName: \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.492885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.492484 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.502084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.493492 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.502084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.493540 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:00.502084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.497895 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.492771 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.503206 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlhh\" (UniqueName: \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.503557 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx7ks\" (UniqueName: \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.503665 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtls\" (UniqueName: \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.503707 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.504127 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.505216 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.505380 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:00.506135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.505847 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.506687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.506362 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.506687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.506437 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.506687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.506487 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.506687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.506539 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.522096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.507538 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:00.522096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.519298 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:00.522096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.519429 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.539673 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.528788 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476k4\" (UniqueName: \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\") pod \"maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.539673 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.533589 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:00.540478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.540453 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwgrk\" (UniqueName: \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.543026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.542817 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:00.556825 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.556762 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:00.571042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.570125 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608282 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608376 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608423 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608467 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608526 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4n66\" (UniqueName: \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.608707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.608585 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkxw\" (UniqueName: \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609311 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609747 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609867 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlhh\" (UniqueName: \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609903 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx7ks\" (UniqueName: \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609941 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtls\" (UniqueName: \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609968 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.610743 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.611697 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.609316 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.616267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.613780 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:00.622117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.616356 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 08:00:00.622820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.622512 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.622820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.622780 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.638239 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.638187 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:00.638887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.638833 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4n66\" (UniqueName: \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.648597 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.648527 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlhh\" (UniqueName: \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\") pod \"maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.659445 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.654293 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtls\" (UniqueName: \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\") pod \"maas-api-revoke-keys-e2e-inf-b-268576-5lzmg\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.659445 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.654935 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkxw\" (UniqueName: \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.664833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.660076 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:00.666571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.666520 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx7ks\" (UniqueName: \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\") pod \"maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.711705 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.711652 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnmc\" (UniqueName: \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.712407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.711945 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk8kv\" (UniqueName: \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.712407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.712107 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.712407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.712146 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.731540 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.731031 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:00.735127 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.735094 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:00.800208 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.799782 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813133 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\") pod \"8c9ef108-46fa-458b-af70-90065ca8c203\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813223 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\" (UniqueName: \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\") pod \"8c9ef108-46fa-458b-af70-90065ca8c203\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813451 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\") pod \"8c9ef108-46fa-458b-af70-90065ca8c203\" (UID: \"8c9ef108-46fa-458b-af70-90065ca8c203\") " Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813726 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813767 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.813856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnmc\" (UniqueName: \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.814812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.814088 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk8kv\" (UniqueName: \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.818407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.815461 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.818407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.816853 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.818407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.817153 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca" pod "8c9ef108-46fa-458b-af70-90065ca8c203" (UID: "8c9ef108-46fa-458b-af70-90065ca8c203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:00.820135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.819468 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:00.820259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.820239 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp" pod "8c9ef108-46fa-458b-af70-90065ca8c203" (UID: "8c9ef108-46fa-458b-af70-90065ca8c203"). InnerVolumeSpecName "kube-api-access-fztnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:00.822146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.820578 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca" pod "8c9ef108-46fa-458b-af70-90065ca8c203" (UID: "8c9ef108-46fa-458b-af70-90065ca8c203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:00.841113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.841042 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk8kv\" (UniqueName: \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.843807 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.843554 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnmc\" (UniqueName: \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\") pod \"maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.864939 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.864679 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:00.895683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.885410 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:00.895683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.886378 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:00.924699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.923451 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:00.925530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.925444 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:00.925530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.925491 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fztnp\" (UniqueName: \"kubernetes.io/projected/8c9ef108-46fa-458b-af70-90065ca8c203-kube-api-access-fztnp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:00.925530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:00.925510 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9ef108-46fa-458b-af70-90065ca8c203-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:01.305624 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.305600 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" Oct 30 08:00:01.306064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.305674 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn" event={"ID":"8c9ef108-46fa-458b-af70-90065ca8c203","Type":"ContainerDied","Data":"fa80849176e0f30d757f5ddaf919b027bfb53b42b0af4c79b02c783544106478"} Oct 30 08:00:01.306064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.305730 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa80849176e0f30d757f5ddaf919b027bfb53b42b0af4c79b02c783544106478" Oct 30 08:00:01.306697 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.306671 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eced14c_1e20_4a1e_bc1b_bc705c0dc0de.slice/crio-336dee75427daf3ef26bef80bc27b8c9e38f54a436df8bc209fa818e58380b0f WatchSource:0}: Error finding container 336dee75427daf3ef26bef80bc27b8c9e38f54a436df8bc209fa818e58380b0f: Status 404 returned error can't find the container with id 336dee75427daf3ef26bef80bc27b8c9e38f54a436df8bc209fa818e58380b0f Oct 30 08:00:01.308200 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.308175 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:01.338936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.338887 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:01.351627 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.351602 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:01.378698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.378654 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:01.431727 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.431693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:01.431951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.431860 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:01.437831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.437767 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:01.456478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.452378 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:01.493108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.492874 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:01.503752 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.503700 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:01.527308 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.527268 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:01.568893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.560396 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:01.568893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.566710 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:01.568893 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.567148 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad9b3c2_06a2_4f2b_b995_69fbfbf1b1d5.slice/crio-9016f9683ae5b082eb88ddb3cec102a54ec6f8790230528902b90d383d16ba8e WatchSource:0}: Error finding container 9016f9683ae5b082eb88ddb3cec102a54ec6f8790230528902b90d383d16ba8e: Status 404 returned error can't find the container with id 9016f9683ae5b082eb88ddb3cec102a54ec6f8790230528902b90d383d16ba8e Oct 30 08:00:01.570588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.570559 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:01.705746 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.705710 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg"] Oct 30 08:00:01.730460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.730430 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:01.747617 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.747589 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 08:00:01.762653 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.762326 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 08:00:01.777249 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.777227 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:01.781031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:01.781008 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:01.894563 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.894528 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc6cbd5_f519_4d14_82f1_f2e0eaf123cf.slice/crio-4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593 WatchSource:0}: Error finding container 4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593: Status 404 returned error can't find the container with id 4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593 Oct 30 08:00:01.895142 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.894988 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711294e8_5614_4075_8ab9_5898ca529105.slice/crio-71a77e07f1e4ca934f3257cc9532b89630112cd77ab3f2d6aad4491f5b3086d9 WatchSource:0}: Error finding container 71a77e07f1e4ca934f3257cc9532b89630112cd77ab3f2d6aad4491f5b3086d9: Status 404 returned error can't find the container with id 71a77e07f1e4ca934f3257cc9532b89630112cd77ab3f2d6aad4491f5b3086d9 Oct 30 08:00:01.895733 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.895709 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2feccda1_6ff1_4e55_b1ae_28e499ac0f42.slice/crio-e21e722406ea728106a4931d274e24a10ae42e23b6fdc96a6939c0b5240923f1 WatchSource:0}: Error finding container e21e722406ea728106a4931d274e24a10ae42e23b6fdc96a6939c0b5240923f1: Status 404 returned error can't find the container with id e21e722406ea728106a4931d274e24a10ae42e23b6fdc96a6939c0b5240923f1 Oct 30 08:00:01.895972 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:01.895946 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa98e977_1e85_4da0_92e2_55e96732b578.slice/crio-2ff3d5ed4788f09835a1fe55909118644aa8fda40e10a03c910fa77d91564f9e WatchSource:0}: Error finding container 2ff3d5ed4788f09835a1fe55909118644aa8fda40e10a03c910fa77d91564f9e: Status 404 returned error can't find the container with id 2ff3d5ed4788f09835a1fe55909118644aa8fda40e10a03c910fa77d91564f9e Oct 30 08:00:02.316911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.316871 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerStarted","Data":"9016f9683ae5b082eb88ddb3cec102a54ec6f8790230528902b90d383d16ba8e"} Oct 30 08:00:02.318296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.318264 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerStarted","Data":"17abc25184e46d446f2d35f37714e25d085eeed3600f3b44fd94ef3387eebc41"} Oct 30 08:00:02.319561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.319538 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerStarted","Data":"e21e722406ea728106a4931d274e24a10ae42e23b6fdc96a6939c0b5240923f1"} Oct 30 08:00:02.320842 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.320817 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" event={"ID":"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf","Type":"ContainerStarted","Data":"4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593"} Oct 30 08:00:02.322132 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.322058 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerStarted","Data":"71a77e07f1e4ca934f3257cc9532b89630112cd77ab3f2d6aad4491f5b3086d9"} Oct 30 08:00:02.323874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.323850 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerStarted","Data":"c242a296c220c423b63b88d737263ee69897d4153b597d4b9736275fdb4d493a"} Oct 30 08:00:02.323975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.323890 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerStarted","Data":"731f0bd4dcdbb5d01a810425cd33579fac6da08ce7b9cd5537bf236fe4f0de26"} Oct 30 08:00:02.325250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.325226 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerStarted","Data":"83dde05ec8a4b176f7437265bc045d68a06ac9a41a3611d15e235587f589bb4d"} Oct 30 08:00:02.326608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.326585 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerStarted","Data":"5b9bed8afbadb6ff690442259e4f911d4f1c2956119cb860b54915fa6bcd26a8"} Oct 30 08:00:02.327873 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.327851 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerStarted","Data":"bce1930f775d52fdd3abf78a21bd0d0f39052875e6f2a0a2de3ca40d2a685ccb"} Oct 30 08:00:02.329104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.329084 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerStarted","Data":"9283715216bf6bc0a1d750d804e774d37a4ae414c77feb1ac75896b346e42048"} Oct 30 08:00:02.330602 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.330580 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerStarted","Data":"44a11f5ae237e6ced9b92119e10a4169eec0d571a2eca82a1e4614d2300cf25f"} Oct 30 08:00:02.330703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.330643 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerStarted","Data":"336dee75427daf3ef26bef80bc27b8c9e38f54a436df8bc209fa818e58380b0f"} Oct 30 08:00:02.331940 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.331917 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" event={"ID":"eaac19a2-19af-431e-9553-4eaea056ed79","Type":"ContainerStarted","Data":"f86e2de77b9b6897dadd1d3d579c5cc3c314a66dcdb67572cddc6b0d4ad703cc"} Oct 30 08:00:02.333181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.333162 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" event={"ID":"1ec31c06-c07b-4d87-852a-cad68487b55b","Type":"ContainerStarted","Data":"1b18898b4600a9a2ae548ac1849448741ab826c92593da5b10f8ac438464ee07"} Oct 30 08:00:02.334416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.334396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerStarted","Data":"b4fc934a44c0d6f70bec93a37a33f70948881a3cfc3c5d3fd5a634ab5e24c7b0"} Oct 30 08:00:02.335619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.335597 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerStarted","Data":"43d55881be58e1fae29c30712913360c4258ebfca0be302edc3cafba2f11ae16"} Oct 30 08:00:02.336859 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.336839 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerStarted","Data":"06ddaea116b1c17eb06afbe20549fa014fa89389a2faaa049b70e6e17634d7a7"} Oct 30 08:00:02.338053 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.338034 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerStarted","Data":"2ff3d5ed4788f09835a1fe55909118644aa8fda40e10a03c910fa77d91564f9e"} Oct 30 08:00:02.349553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.349535 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 08:00:02.498376 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.498307 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" podStartSLOduration=2.4982931649999998 podStartE2EDuration="2.498293165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.414008589 +0000 UTC m=+3766.104926025" watchObservedRunningTime="2026-10-30 08:00:02.498293165 +0000 UTC m=+3766.189210601" Oct 30 08:00:02.498509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:02.498412 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" podStartSLOduration=2.498405143 podStartE2EDuration="2.498405143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:02.496868629 +0000 UTC m=+3766.187786067" watchObservedRunningTime="2026-10-30 08:00:02.498405143 +0000 UTC m=+3766.189322581" Oct 30 08:00:03.353712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.353667 5894 generic.go:356] "Generic (PLEG): container finished" podID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerID="5378084f4fa58f14e20e844757e0d6b990d9d1234e890cb71744aaca23f50cd2" exitCode=60 Oct 30 08:00:03.354209 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.353743 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerDied","Data":"5378084f4fa58f14e20e844757e0d6b990d9d1234e890cb71744aaca23f50cd2"} Oct 30 08:00:03.354209 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.354033 5894 scope.go:122] "RemoveContainer" containerID="5378084f4fa58f14e20e844757e0d6b990d9d1234e890cb71744aaca23f50cd2" Oct 30 08:00:03.370953 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.356038 5894 generic.go:356] "Generic (PLEG): container finished" podID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerID="3dbbed933035a2601d61335b4857ced9319927eec11a378ee20eba3340ab1398" exitCode=60 Oct 30 08:00:03.370953 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.356198 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerDied","Data":"3dbbed933035a2601d61335b4857ced9319927eec11a378ee20eba3340ab1398"} Oct 30 08:00:03.370953 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.356594 5894 scope.go:122] "RemoveContainer" containerID="3dbbed933035a2601d61335b4857ced9319927eec11a378ee20eba3340ab1398" Oct 30 08:00:03.377901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.377866 5894 generic.go:356] "Generic (PLEG): container finished" podID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerID="644d9a36db52d1abb227faf2afc940b875b74641b73b00ae1b33cf4f0f36deb7" exitCode=60 Oct 30 08:00:03.378016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.377963 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerDied","Data":"644d9a36db52d1abb227faf2afc940b875b74641b73b00ae1b33cf4f0f36deb7"} Oct 30 08:00:03.378905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.378761 5894 scope.go:122] "RemoveContainer" containerID="644d9a36db52d1abb227faf2afc940b875b74641b73b00ae1b33cf4f0f36deb7" Oct 30 08:00:03.381364 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.381340 5894 generic.go:356] "Generic (PLEG): container finished" podID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerID="44a11f5ae237e6ced9b92119e10a4169eec0d571a2eca82a1e4614d2300cf25f" exitCode=60 Oct 30 08:00:03.381476 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.381419 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerDied","Data":"44a11f5ae237e6ced9b92119e10a4169eec0d571a2eca82a1e4614d2300cf25f"} Oct 30 08:00:03.381958 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.381940 5894 scope.go:122] "RemoveContainer" containerID="44a11f5ae237e6ced9b92119e10a4169eec0d571a2eca82a1e4614d2300cf25f" Oct 30 08:00:03.386562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.386531 5894 generic.go:356] "Generic (PLEG): container finished" podID="eaac19a2-19af-431e-9553-4eaea056ed79" containerID="da80040cc489a489bee01fd0d5dadca0522954abf6d4b334e9e2f982243f147f" exitCode=60 Oct 30 08:00:03.386698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.386618 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" event={"ID":"eaac19a2-19af-431e-9553-4eaea056ed79","Type":"ContainerDied","Data":"da80040cc489a489bee01fd0d5dadca0522954abf6d4b334e9e2f982243f147f"} Oct 30 08:00:03.387000 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.386976 5894 scope.go:122] "RemoveContainer" containerID="da80040cc489a489bee01fd0d5dadca0522954abf6d4b334e9e2f982243f147f" Oct 30 08:00:03.394756 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.394733 5894 generic.go:356] "Generic (PLEG): container finished" podID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerID="b444b9f6d79155fa51e9d22aefe7a23d9096252aa8be29f515027d0ef5fed617" exitCode=60 Oct 30 08:00:03.395016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.394993 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" event={"ID":"1ec31c06-c07b-4d87-852a-cad68487b55b","Type":"ContainerDied","Data":"b444b9f6d79155fa51e9d22aefe7a23d9096252aa8be29f515027d0ef5fed617"} Oct 30 08:00:03.395874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.395059 5894 scope.go:122] "RemoveContainer" containerID="b444b9f6d79155fa51e9d22aefe7a23d9096252aa8be29f515027d0ef5fed617" Oct 30 08:00:03.397728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.397703 5894 generic.go:356] "Generic (PLEG): container finished" podID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerID="ffad2a4344a20fa394f83dd3e5abb0accf33756c484145ae4b2ce670e627d721" exitCode=60 Oct 30 08:00:03.397850 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.397833 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerDied","Data":"ffad2a4344a20fa394f83dd3e5abb0accf33756c484145ae4b2ce670e627d721"} Oct 30 08:00:03.398591 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.398571 5894 scope.go:122] "RemoveContainer" containerID="ffad2a4344a20fa394f83dd3e5abb0accf33756c484145ae4b2ce670e627d721" Oct 30 08:00:03.401121 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.401093 5894 generic.go:356] "Generic (PLEG): container finished" podID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerID="8458b11f2bdf1f8b5c206efe2cb82844491e8da675f30b05b979195ee1e9c85f" exitCode=60 Oct 30 08:00:03.401498 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.401476 5894 scope.go:122] "RemoveContainer" containerID="8458b11f2bdf1f8b5c206efe2cb82844491e8da675f30b05b979195ee1e9c85f" Oct 30 08:00:03.401914 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.401887 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerDied","Data":"8458b11f2bdf1f8b5c206efe2cb82844491e8da675f30b05b979195ee1e9c85f"} Oct 30 08:00:03.413015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.412922 5894 generic.go:356] "Generic (PLEG): container finished" podID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerID="5fde17afb87520f66ccd0ee8509b3dc2952d6d7eb628cf42f5b4162ba2c0d241" exitCode=60 Oct 30 08:00:03.413254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.413233 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerDied","Data":"5fde17afb87520f66ccd0ee8509b3dc2952d6d7eb628cf42f5b4162ba2c0d241"} Oct 30 08:00:03.413802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.413686 5894 scope.go:122] "RemoveContainer" containerID="5fde17afb87520f66ccd0ee8509b3dc2952d6d7eb628cf42f5b4162ba2c0d241" Oct 30 08:00:03.417704 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.417663 5894 generic.go:356] "Generic (PLEG): container finished" podID="fa98e977-1e85-4da0-92e2-55e96732b578" containerID="1a60c4093ac7c7a9a8e672474f8c1bd498e5511a75d1082959a9d306249e4c72" exitCode=60 Oct 30 08:00:03.417893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.417728 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerDied","Data":"1a60c4093ac7c7a9a8e672474f8c1bd498e5511a75d1082959a9d306249e4c72"} Oct 30 08:00:03.418298 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.418001 5894 scope.go:122] "RemoveContainer" containerID="1a60c4093ac7c7a9a8e672474f8c1bd498e5511a75d1082959a9d306249e4c72" Oct 30 08:00:03.420135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.420109 5894 generic.go:356] "Generic (PLEG): container finished" podID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerID="5388563955d4d98becc2af67d274c69ffb3f5578b69585f523dc9d69b4706cb4" exitCode=60 Oct 30 08:00:03.420387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.420363 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerDied","Data":"5388563955d4d98becc2af67d274c69ffb3f5578b69585f523dc9d69b4706cb4"} Oct 30 08:00:03.420513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.420497 5894 scope.go:122] "RemoveContainer" containerID="5388563955d4d98becc2af67d274c69ffb3f5578b69585f523dc9d69b4706cb4" Oct 30 08:00:03.422876 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.422854 5894 generic.go:356] "Generic (PLEG): container finished" podID="01e99876-ea97-489e-9728-25fa00c726d6" containerID="18382df5c899d443f6547c154814346a98b1e6449af0e8e2255c102d1a203465" exitCode=60 Oct 30 08:00:03.422972 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.422928 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerDied","Data":"18382df5c899d443f6547c154814346a98b1e6449af0e8e2255c102d1a203465"} Oct 30 08:00:03.423161 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.423147 5894 scope.go:122] "RemoveContainer" containerID="18382df5c899d443f6547c154814346a98b1e6449af0e8e2255c102d1a203465" Oct 30 08:00:03.425692 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.425664 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerStarted","Data":"760b701179888a91cefa65f76de04d56fda6d1f9911a16ffb7c9d991b58fbb31"} Oct 30 08:00:03.425935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.425919 5894 scope.go:122] "RemoveContainer" containerID="760b701179888a91cefa65f76de04d56fda6d1f9911a16ffb7c9d991b58fbb31" Oct 30 08:00:03.427384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.427365 5894 generic.go:356] "Generic (PLEG): container finished" podID="adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" containerID="3d1fe9c5f7ff61959c752b873b024e37b0cbbc95a595ecf78458ba0627e6c5aa" exitCode=0 Oct 30 08:00:03.427473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.427462 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" event={"ID":"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf","Type":"ContainerDied","Data":"3d1fe9c5f7ff61959c752b873b024e37b0cbbc95a595ecf78458ba0627e6c5aa"} Oct 30 08:00:03.432250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.432228 5894 generic.go:356] "Generic (PLEG): container finished" podID="711294e8-5614-4075-8ab9-5898ca529105" containerID="5d02968a2d330c9e7b3821f4e354827812ea6bf60a27f0486b4c14be2058a4f6" exitCode=60 Oct 30 08:00:03.433023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.433005 5894 scope.go:122] "RemoveContainer" containerID="5d02968a2d330c9e7b3821f4e354827812ea6bf60a27f0486b4c14be2058a4f6" Oct 30 08:00:03.433470 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.433445 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerDied","Data":"5d02968a2d330c9e7b3821f4e354827812ea6bf60a27f0486b4c14be2058a4f6"} Oct 30 08:00:03.440503 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.439777 5894 generic.go:356] "Generic (PLEG): container finished" podID="1c541641-0030-4fa0-8a8d-516668265dc3" containerID="c242a296c220c423b63b88d737263ee69897d4153b597d4b9736275fdb4d493a" exitCode=60 Oct 30 08:00:03.440503 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.439858 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerDied","Data":"c242a296c220c423b63b88d737263ee69897d4153b597d4b9736275fdb4d493a"} Oct 30 08:00:03.440503 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.440164 5894 scope.go:122] "RemoveContainer" containerID="c242a296c220c423b63b88d737263ee69897d4153b597d4b9736275fdb4d493a" Oct 30 08:00:03.448877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.448590 5894 generic.go:356] "Generic (PLEG): container finished" podID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerID="6861deebf789153a0f56bc42aa4084dddb155951d7f2e81912dabf6f101b0ea1" exitCode=60 Oct 30 08:00:03.450964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.450942 5894 scope.go:122] "RemoveContainer" containerID="6861deebf789153a0f56bc42aa4084dddb155951d7f2e81912dabf6f101b0ea1" Oct 30 08:00:03.451315 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.451293 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerDied","Data":"6861deebf789153a0f56bc42aa4084dddb155951d7f2e81912dabf6f101b0ea1"} Oct 30 08:00:03.730110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:03.730074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:04.459397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.459362 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerStarted","Data":"0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e"} Oct 30 08:00:04.461487 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.461458 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerStarted","Data":"d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4"} Oct 30 08:00:04.463649 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.463610 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerStarted","Data":"99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c"} Oct 30 08:00:04.465212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.465190 5894 generic.go:356] "Generic (PLEG): container finished" podID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerID="760b701179888a91cefa65f76de04d56fda6d1f9911a16ffb7c9d991b58fbb31" exitCode=60 Oct 30 08:00:04.465296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.465262 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerDied","Data":"760b701179888a91cefa65f76de04d56fda6d1f9911a16ffb7c9d991b58fbb31"} Oct 30 08:00:04.467117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.467091 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerStarted","Data":"d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7"} Oct 30 08:00:04.469296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.469271 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerStarted","Data":"64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b"} Oct 30 08:00:04.507732 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.505624 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" podStartSLOduration=4.505578723 podStartE2EDuration="4.505578723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.503115529 +0000 UTC m=+3768.194032968" watchObservedRunningTime="2026-10-30 08:00:04.505578723 +0000 UTC m=+3768.196496161" Oct 30 08:00:04.547498 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.547439 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" podStartSLOduration=4.547419516 podStartE2EDuration="4.547419516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.546226555 +0000 UTC m=+3768.237143993" watchObservedRunningTime="2026-10-30 08:00:04.547419516 +0000 UTC m=+3768.238336953" Oct 30 08:00:04.652986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.652896 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" podStartSLOduration=4.652880027 podStartE2EDuration="4.652880027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.610098755 +0000 UTC m=+3768.301016207" watchObservedRunningTime="2026-10-30 08:00:04.652880027 +0000 UTC m=+3768.343797467" Oct 30 08:00:04.655609 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.655572 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" podStartSLOduration=4.65555934 podStartE2EDuration="4.65555934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.651939521 +0000 UTC m=+3768.342856959" watchObservedRunningTime="2026-10-30 08:00:04.65555934 +0000 UTC m=+3768.346476778" Oct 30 08:00:04.705133 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.705082 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" podStartSLOduration=4.705066513 podStartE2EDuration="4.705066513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:04.703906001 +0000 UTC m=+3768.394823439" watchObservedRunningTime="2026-10-30 08:00:04.705066513 +0000 UTC m=+3768.395983950" Oct 30 08:00:04.723706 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.723669 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:04.817800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.816451 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\" (UniqueName: \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\") pod \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " Oct 30 08:00:04.817800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.816567 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\") pod \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " Oct 30 08:00:04.821658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.821567 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\") pod \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\" (UID: \"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf\") " Oct 30 08:00:04.830134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.830083 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls" pod "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" (UID: "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf"). InnerVolumeSpecName "kube-api-access-lbtls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:04.830506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.830459 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca" pod "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" (UID: "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:04.836084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.836028 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca" pod "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" (UID: "adc6cbd5-f519-4d14-82f1-f2e0eaf123cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:04.857199 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:04.857146 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:00:04.923766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.923723 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:04.923766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.923762 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:04.923937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:04.923777 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtls\" (UniqueName: \"kubernetes.io/projected/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf-kube-api-access-lbtls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:05.479864 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.479828 5894 generic.go:356] "Generic (PLEG): container finished" podID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerID="06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb" exitCode=60 Oct 30 08:00:05.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.479883 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerDied","Data":"06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb"} Oct 30 08:00:05.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.479944 5894 scope.go:122] "RemoveContainer" containerID="8458b11f2bdf1f8b5c206efe2cb82844491e8da675f30b05b979195ee1e9c85f" Oct 30 08:00:05.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.480278 5894 scope.go:122] "RemoveContainer" containerID="06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb" Oct 30 08:00:05.480682 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.480652 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254_odh-ai-gateway-infra(abbe9d7a-6ea5-4ca5-8d46-863cec9cedea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" Oct 30 08:00:05.482219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.482192 5894 generic.go:356] "Generic (PLEG): container finished" podID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerID="d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4" exitCode=60 Oct 30 08:00:05.482362 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.482228 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerDied","Data":"d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4"} Oct 30 08:00:05.482570 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.482553 5894 scope.go:122] "RemoveContainer" containerID="d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4" Oct 30 08:00:05.482944 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.482921 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj_odh-ai-gateway-infra(9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" Oct 30 08:00:05.484390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.484366 5894 generic.go:356] "Generic (PLEG): container finished" podID="fa98e977-1e85-4da0-92e2-55e96732b578" containerID="99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c" exitCode=60 Oct 30 08:00:05.484495 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.484446 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerDied","Data":"99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c"} Oct 30 08:00:05.484747 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.484731 5894 scope.go:122] "RemoveContainer" containerID="99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c" Oct 30 08:00:05.485096 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.485073 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5_odh-ai-gateway-infra(fa98e977-1e85-4da0-92e2-55e96732b578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" Oct 30 08:00:05.486700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.486678 5894 generic.go:356] "Generic (PLEG): container finished" podID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerID="c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb" exitCode=60 Oct 30 08:00:05.486787 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.486759 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerDied","Data":"c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb"} Oct 30 08:00:05.487132 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.487091 5894 scope.go:122] "RemoveContainer" containerID="c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb" Oct 30 08:00:05.487453 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.487427 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9_odh-ai-gateway-infra(9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" Oct 30 08:00:05.488725 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.488700 5894 generic.go:356] "Generic (PLEG): container finished" podID="01e99876-ea97-489e-9728-25fa00c726d6" containerID="916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc" exitCode=60 Oct 30 08:00:05.488801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.488780 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerDied","Data":"916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc"} Oct 30 08:00:05.489002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.488989 5894 scope.go:122] "RemoveContainer" containerID="916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc" Oct 30 08:00:05.489291 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.489272 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm_odh-ai-gateway-infra(01e99876-ea97-489e-9728-25fa00c726d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" podUID="01e99876-ea97-489e-9728-25fa00c726d6" Oct 30 08:00:05.490544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.490524 5894 generic.go:356] "Generic (PLEG): container finished" podID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerID="87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be" exitCode=60 Oct 30 08:00:05.490668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.490553 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerDied","Data":"87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be"} Oct 30 08:00:05.490846 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.490833 5894 scope.go:122] "RemoveContainer" containerID="87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be" Oct 30 08:00:05.491114 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.491095 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp_odh-ai-gateway-infra(2feccda1-6ff1-4e55-b1ae-28e499ac0f42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" Oct 30 08:00:05.491942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.491925 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" Oct 30 08:00:05.492018 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.491954 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg" event={"ID":"adc6cbd5-f519-4d14-82f1-f2e0eaf123cf","Type":"ContainerDied","Data":"4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593"} Oct 30 08:00:05.492018 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.491998 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4745052bd75d6f8d2dc723d616cc01926e85ecf9ee726b79f688ba6145a2e593" Oct 30 08:00:05.494882 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.494861 5894 generic.go:356] "Generic (PLEG): container finished" podID="711294e8-5614-4075-8ab9-5898ca529105" containerID="d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7" exitCode=60 Oct 30 08:00:05.495020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.494986 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerDied","Data":"d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7"} Oct 30 08:00:05.495177 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.495159 5894 scope.go:122] "RemoveContainer" containerID="d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7" Oct 30 08:00:05.495453 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.495430 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl_odh-ai-gateway-infra(711294e8-5614-4075-8ab9-5898ca529105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" podUID="711294e8-5614-4075-8ab9-5898ca529105" Oct 30 08:00:05.497024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.497001 5894 generic.go:356] "Generic (PLEG): container finished" podID="1c541641-0030-4fa0-8a8d-516668265dc3" containerID="732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8" exitCode=60 Oct 30 08:00:05.497133 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.497082 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerDied","Data":"732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8"} Oct 30 08:00:05.497325 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.497310 5894 scope.go:122] "RemoveContainer" containerID="732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8" Oct 30 08:00:05.497554 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.497536 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh_odh-ai-gateway-infra(1c541641-0030-4fa0-8a8d-516668265dc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" Oct 30 08:00:05.499135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.499113 5894 generic.go:356] "Generic (PLEG): container finished" podID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerID="ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5" exitCode=60 Oct 30 08:00:05.499233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.499181 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerDied","Data":"ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5"} Oct 30 08:00:05.499579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.499538 5894 scope.go:122] "RemoveContainer" containerID="ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5" Oct 30 08:00:05.499836 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.499815 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n_odh-ai-gateway-infra(ef6b14ef-4cdd-4afd-9668-abfd139020d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" Oct 30 08:00:05.501371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.501346 5894 generic.go:356] "Generic (PLEG): container finished" podID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerID="64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b" exitCode=60 Oct 30 08:00:05.501463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.501431 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerDied","Data":"64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b"} Oct 30 08:00:05.501650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.501620 5894 scope.go:122] "RemoveContainer" containerID="64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b" Oct 30 08:00:05.501908 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.501885 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n_odh-ai-gateway-infra(5c3c9121-b3c1-45e1-97d8-393910981f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" Oct 30 08:00:05.503373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.503348 5894 generic.go:356] "Generic (PLEG): container finished" podID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerID="063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202" exitCode=60 Oct 30 08:00:05.503456 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.503388 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerDied","Data":"063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202"} Oct 30 08:00:05.503679 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.503652 5894 scope.go:122] "RemoveContainer" containerID="063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202" Oct 30 08:00:05.503905 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.503880 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g_odh-ai-gateway-infra(9a600d24-d431-4166-a41c-4f3c3012d5ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" Oct 30 08:00:05.505343 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.505323 5894 generic.go:356] "Generic (PLEG): container finished" podID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerID="f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4" exitCode=60 Oct 30 08:00:05.505443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.505405 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerDied","Data":"f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4"} Oct 30 08:00:05.505592 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.505578 5894 scope.go:122] "RemoveContainer" containerID="f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4" Oct 30 08:00:05.505837 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.505815 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4_odh-ai-gateway-infra(09c415f0-2dcb-4da0-b3eb-f26abdb53126)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" Oct 30 08:00:05.507337 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.507317 5894 generic.go:356] "Generic (PLEG): container finished" podID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerID="1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d" exitCode=60 Oct 30 08:00:05.507434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.507396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerDied","Data":"1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d"} Oct 30 08:00:05.507615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.507598 5894 scope.go:122] "RemoveContainer" containerID="1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d" Oct 30 08:00:05.507874 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.507851 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts_odh-ai-gateway-infra(3eced14c-1e20-4a1e-bc1b-bc705c0dc0de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" Oct 30 08:00:05.509341 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.509321 5894 generic.go:356] "Generic (PLEG): container finished" podID="eaac19a2-19af-431e-9553-4eaea056ed79" containerID="0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf" exitCode=60 Oct 30 08:00:05.509452 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.509380 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" event={"ID":"eaac19a2-19af-431e-9553-4eaea056ed79","Type":"ContainerDied","Data":"0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf"} Oct 30 08:00:05.509621 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.509605 5894 scope.go:122] "RemoveContainer" containerID="0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf" Oct 30 08:00:05.509879 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.509856 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz_odh-ai-gateway-infra(eaac19a2-19af-431e-9553-4eaea056ed79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" Oct 30 08:00:05.511245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.511223 5894 generic.go:356] "Generic (PLEG): container finished" podID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerID="6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561" exitCode=60 Oct 30 08:00:05.511371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.511303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" event={"ID":"1ec31c06-c07b-4d87-852a-cad68487b55b","Type":"ContainerDied","Data":"6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561"} Oct 30 08:00:05.511533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.511519 5894 scope.go:122] "RemoveContainer" containerID="6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561" Oct 30 08:00:05.511780 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.511757 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl_odh-ai-gateway-infra(1ec31c06-c07b-4d87-852a-cad68487b55b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" Oct 30 08:00:05.513192 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.513172 5894 generic.go:356] "Generic (PLEG): container finished" podID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerID="0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e" exitCode=60 Oct 30 08:00:05.513276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.513221 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerDied","Data":"0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e"} Oct 30 08:00:05.513409 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.513398 5894 scope.go:122] "RemoveContainer" containerID="0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e" Oct 30 08:00:05.513591 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:05.513577 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9_odh-ai-gateway-infra(cfa7e21f-3373-4018-ae96-582ce2ff9f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" Oct 30 08:00:05.532573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.532542 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:05.667213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.667190 5894 scope.go:122] "RemoveContainer" containerID="5fde17afb87520f66ccd0ee8509b3dc2952d6d7eb628cf42f5b4162ba2c0d241" Oct 30 08:00:05.682564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.682537 5894 scope.go:122] "RemoveContainer" containerID="1a60c4093ac7c7a9a8e672474f8c1bd498e5511a75d1082959a9d306249e4c72" Oct 30 08:00:05.696911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.696853 5894 scope.go:122] "RemoveContainer" containerID="5388563955d4d98becc2af67d274c69ffb3f5578b69585f523dc9d69b4706cb4" Oct 30 08:00:05.712821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.712800 5894 scope.go:122] "RemoveContainer" containerID="18382df5c899d443f6547c154814346a98b1e6449af0e8e2255c102d1a203465" Oct 30 08:00:05.728377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.728357 5894 scope.go:122] "RemoveContainer" containerID="760b701179888a91cefa65f76de04d56fda6d1f9911a16ffb7c9d991b58fbb31" Oct 30 08:00:05.743527 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.743504 5894 scope.go:122] "RemoveContainer" containerID="5d02968a2d330c9e7b3821f4e354827812ea6bf60a27f0486b4c14be2058a4f6" Oct 30 08:00:05.758743 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.758714 5894 scope.go:122] "RemoveContainer" containerID="c242a296c220c423b63b88d737263ee69897d4153b597d4b9736275fdb4d493a" Oct 30 08:00:05.774370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.774348 5894 scope.go:122] "RemoveContainer" containerID="6861deebf789153a0f56bc42aa4084dddb155951d7f2e81912dabf6f101b0ea1" Oct 30 08:00:05.793700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.793669 5894 scope.go:122] "RemoveContainer" containerID="5378084f4fa58f14e20e844757e0d6b990d9d1234e890cb71744aaca23f50cd2" Oct 30 08:00:05.810104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.810080 5894 scope.go:122] "RemoveContainer" containerID="3dbbed933035a2601d61335b4857ced9319927eec11a378ee20eba3340ab1398" Oct 30 08:00:05.825008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.824977 5894 scope.go:122] "RemoveContainer" containerID="644d9a36db52d1abb227faf2afc940b875b74641b73b00ae1b33cf4f0f36deb7" Oct 30 08:00:05.841004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.840981 5894 scope.go:122] "RemoveContainer" containerID="44a11f5ae237e6ced9b92119e10a4169eec0d571a2eca82a1e4614d2300cf25f" Oct 30 08:00:05.857540 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.857518 5894 scope.go:122] "RemoveContainer" containerID="da80040cc489a489bee01fd0d5dadca0522954abf6d4b334e9e2f982243f147f" Oct 30 08:00:05.872135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.872111 5894 scope.go:122] "RemoveContainer" containerID="b444b9f6d79155fa51e9d22aefe7a23d9096252aa8be29f515027d0ef5fed617" Oct 30 08:00:05.887819 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:05.887793 5894 scope.go:122] "RemoveContainer" containerID="ffad2a4344a20fa394f83dd3e5abb0accf33756c484145ae4b2ce670e627d721" Oct 30 08:00:06.523694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.523664 5894 scope.go:122] "RemoveContainer" containerID="0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e" Oct 30 08:00:06.524176 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.524120 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9_odh-ai-gateway-infra(cfa7e21f-3373-4018-ae96-582ce2ff9f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" Oct 30 08:00:06.525542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.525518 5894 scope.go:122] "RemoveContainer" containerID="06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb" Oct 30 08:00:06.525902 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.525880 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254_odh-ai-gateway-infra(abbe9d7a-6ea5-4ca5-8d46-863cec9cedea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" Oct 30 08:00:06.527230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.527212 5894 scope.go:122] "RemoveContainer" containerID="d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4" Oct 30 08:00:06.527491 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.527470 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj_odh-ai-gateway-infra(9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" Oct 30 08:00:06.528863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.528844 5894 scope.go:122] "RemoveContainer" containerID="99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c" Oct 30 08:00:06.529110 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.529086 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5_odh-ai-gateway-infra(fa98e977-1e85-4da0-92e2-55e96732b578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" Oct 30 08:00:06.530690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.530668 5894 scope.go:122] "RemoveContainer" containerID="c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb" Oct 30 08:00:06.530915 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.530896 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9_odh-ai-gateway-infra(9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" Oct 30 08:00:06.532371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.532350 5894 scope.go:122] "RemoveContainer" containerID="916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc" Oct 30 08:00:06.532587 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.532565 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm_odh-ai-gateway-infra(01e99876-ea97-489e-9728-25fa00c726d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" podUID="01e99876-ea97-489e-9728-25fa00c726d6" Oct 30 08:00:06.533940 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.533923 5894 scope.go:122] "RemoveContainer" containerID="87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be" Oct 30 08:00:06.534155 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.534131 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp_odh-ai-gateway-infra(2feccda1-6ff1-4e55-b1ae-28e499ac0f42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" Oct 30 08:00:06.535615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.535592 5894 scope.go:122] "RemoveContainer" containerID="d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7" Oct 30 08:00:06.535858 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.535836 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl_odh-ai-gateway-infra(711294e8-5614-4075-8ab9-5898ca529105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" podUID="711294e8-5614-4075-8ab9-5898ca529105" Oct 30 08:00:06.537356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.537339 5894 scope.go:122] "RemoveContainer" containerID="732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8" Oct 30 08:00:06.537584 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.537559 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh_odh-ai-gateway-infra(1c541641-0030-4fa0-8a8d-516668265dc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" Oct 30 08:00:06.539051 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.539024 5894 scope.go:122] "RemoveContainer" containerID="ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5" Oct 30 08:00:06.539296 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.539260 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n_odh-ai-gateway-infra(ef6b14ef-4cdd-4afd-9668-abfd139020d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" Oct 30 08:00:06.540759 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.540738 5894 scope.go:122] "RemoveContainer" containerID="64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b" Oct 30 08:00:06.540972 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.540950 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n_odh-ai-gateway-infra(5c3c9121-b3c1-45e1-97d8-393910981f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" Oct 30 08:00:06.542577 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.542554 5894 scope.go:122] "RemoveContainer" containerID="063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202" Oct 30 08:00:06.542834 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.542813 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g_odh-ai-gateway-infra(9a600d24-d431-4166-a41c-4f3c3012d5ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" Oct 30 08:00:06.544387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.544366 5894 scope.go:122] "RemoveContainer" containerID="f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4" Oct 30 08:00:06.544595 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.544569 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4_odh-ai-gateway-infra(09c415f0-2dcb-4da0-b3eb-f26abdb53126)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" Oct 30 08:00:06.546138 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.546119 5894 scope.go:122] "RemoveContainer" containerID="1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d" Oct 30 08:00:06.546351 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.546333 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts_odh-ai-gateway-infra(3eced14c-1e20-4a1e-bc1b-bc705c0dc0de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" Oct 30 08:00:06.547793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.547767 5894 scope.go:122] "RemoveContainer" containerID="0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf" Oct 30 08:00:06.547988 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.547970 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz_odh-ai-gateway-infra(eaac19a2-19af-431e-9553-4eaea056ed79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" Oct 30 08:00:06.549324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.549308 5894 scope.go:122] "RemoveContainer" containerID="6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561" Oct 30 08:00:06.549490 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:06.549474 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl_odh-ai-gateway-infra(1ec31c06-c07b-4d87-852a-cad68487b55b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" Oct 30 08:00:06.619371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.619339 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 08:00:06.619547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:06.619526 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" containerID="cri-o://377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144" gracePeriod=30 Oct 30 08:00:07.434179 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:07.434142 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:07.549387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:07.549346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:07.676613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:07.676565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:08.992774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:08.992730 5894 patch_prober.go:29] interesting pod/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:00:08.993189 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:08.992795 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:00:09.194422 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:09.194381 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:10.770918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:10.770876 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 08:00:10.790233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:10.790195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:10.803208 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:10.803167 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 08:00:10.929824 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:10.929779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:11.053588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:11.053545 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:11.903370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:11.903279 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 08:00:12.055979 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.055935 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056061 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056088 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056167 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056209 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056296 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056324 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056388 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:12.056537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056455 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056466 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:12.056537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056511 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\") pod \"ff106477-2682-4035-bfda-adffc03ed0c7\" (UID: \"ff106477-2682-4035-bfda-adffc03ed0c7\") " Oct 30 08:00:12.056799 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056582 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:12.056957 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.056935 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057082 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057190 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057215 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057224 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057234 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff106477-2682-4035-bfda-adffc03ed0c7-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.057261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.057242 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.058797 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.058776 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:00:12.058887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.058867 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "kube-api-access-xgbmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:12.059074 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.059049 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:12.059118 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.059050 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo" pod "ff106477-2682-4035-bfda-adffc03ed0c7" (UID: "ff106477-2682-4035-bfda-adffc03ed0c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:00:12.158382 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.158295 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.158382 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.158325 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff106477-2682-4035-bfda-adffc03ed0c7-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.158382 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.158335 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff106477-2682-4035-bfda-adffc03ed0c7-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.158382 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.158348 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgbmf\" (UniqueName: \"kubernetes.io/projected/ff106477-2682-4035-bfda-adffc03ed0c7-kube-api-access-xgbmf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:12.548424 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.548383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:12.619222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.619182 5894 generic.go:356] "Generic (PLEG): container finished" podID="ff106477-2682-4035-bfda-adffc03ed0c7" containerID="377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144" exitCode=0 Oct 30 08:00:12.619505 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.619243 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" event={"ID":"ff106477-2682-4035-bfda-adffc03ed0c7","Type":"ContainerDied","Data":"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144"} Oct 30 08:00:12.619666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.619597 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" event={"ID":"ff106477-2682-4035-bfda-adffc03ed0c7","Type":"ContainerDied","Data":"eebb4a23b0c82a0e27a0aee7bde6d39806334e27491d4f46ab41c66f28e28b41"} Oct 30 08:00:12.619731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.619707 5894 scope.go:122] "RemoveContainer" containerID="377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144" Oct 30 08:00:12.619772 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.619300 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz" Oct 30 08:00:12.637911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.637830 5894 scope.go:122] "RemoveContainer" containerID="377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144" Oct 30 08:00:12.638383 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:12.638342 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144\": container with ID starting with 377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144 not found: ID does not exist" containerID="377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144" Oct 30 08:00:12.638383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.638377 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144"} err="failed to get container status \"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144\": rpc error: code = NotFound desc = could not find container \"377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144\": container with ID starting with 377eb52f1ca51c92731d0b63d88f0408dc7f9eadbe881e212b139c77822c7144 not found: ID does not exist" Oct 30 08:00:12.687693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.687656 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 08:00:12.699941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:12.699846 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e5fa2d-openshift-default-68d8df7795-dklbz"] Oct 30 08:00:13.159974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:13.159938 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:00:14.412171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:14.412132 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:14.434658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:14.434597 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" path="/var/lib/kubelet/pods/ff106477-2682-4035-bfda-adffc03ed0c7/volumes" Oct 30 08:00:15.771980 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:15.771921 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" podUID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" containerName="payload-processing" containerID="cri-o://4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9" gracePeriod=25 Oct 30 08:00:15.804151 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:15.804115 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" containerID="cri-o://7c231430f8df26fc70948574e883499113e624bf0e6e9f9989748434effefdb1" gracePeriod=25 Oct 30 08:00:16.079663 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.079602 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 08:00:16.111936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.111901 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\" (UniqueName: \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\") pod \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " Oct 30 08:00:16.112149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.112084 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\") pod \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\" (UID: \"7423ac5b-14cf-4d34-a97a-f0156336a7c6\") " Oct 30 08:00:16.112558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.112527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume" pod "7423ac5b-14cf-4d34-a97a-f0156336a7c6" (UID: "7423ac5b-14cf-4d34-a97a-f0156336a7c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:16.112738 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.112721 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7423ac5b-14cf-4d34-a97a-f0156336a7c6-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:16.114194 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.114166 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr" pod "7423ac5b-14cf-4d34-a97a-f0156336a7c6" (UID: "7423ac5b-14cf-4d34-a97a-f0156336a7c6"). InnerVolumeSpecName "kube-api-access-nd2dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:16.213616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.213575 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2dr\" (UniqueName: \"kubernetes.io/projected/7423ac5b-14cf-4d34-a97a-f0156336a7c6-kube-api-access-nd2dr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:16.223256 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.223224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:16.429613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.429521 5894 scope.go:122] "RemoveContainer" containerID="f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4" Oct 30 08:00:16.429821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.429748 5894 scope.go:122] "RemoveContainer" containerID="ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5" Oct 30 08:00:16.429892 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.429876 5894 scope.go:122] "RemoveContainer" containerID="0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e" Oct 30 08:00:16.430029 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.430007 5894 scope.go:122] "RemoveContainer" containerID="87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be" Oct 30 08:00:16.430127 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.430106 5894 scope.go:122] "RemoveContainer" containerID="1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d" Oct 30 08:00:16.430391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.430231 5894 scope.go:122] "RemoveContainer" containerID="0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf" Oct 30 08:00:16.658330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.658295 5894 generic.go:356] "Generic (PLEG): container finished" podID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" containerID="4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9" exitCode=0 Oct 30 08:00:16.658506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.658341 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" Oct 30 08:00:16.658506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.658371 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" event={"ID":"7423ac5b-14cf-4d34-a97a-f0156336a7c6","Type":"ContainerDied","Data":"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9"} Oct 30 08:00:16.658506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.658427 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4" event={"ID":"7423ac5b-14cf-4d34-a97a-f0156336a7c6","Type":"ContainerDied","Data":"8314ee76411d638a296970efbca3096ce4971a827c5f8f74f2aa8f0fabb07157"} Oct 30 08:00:16.658506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.658450 5894 scope.go:122] "RemoveContainer" containerID="4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9" Oct 30 08:00:16.669988 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.669968 5894 scope.go:122] "RemoveContainer" containerID="4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9" Oct 30 08:00:16.670287 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:16.670267 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9\": container with ID starting with 4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9 not found: ID does not exist" containerID="4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9" Oct 30 08:00:16.670332 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.670296 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9"} err="failed to get container status \"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9\": rpc error: code = NotFound desc = could not find container \"4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9\": container with ID starting with 4e832607afc03c318ed66a48599d1bd8b9d6119fd16437ba010e3c515a2d25c9 not found: ID does not exist" Oct 30 08:00:16.713392 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.713360 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 08:00:16.718342 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:16.718316 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-268576-89cdc99d6-2bfm4"] Oct 30 08:00:17.424870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.424839 5894 scope.go:122] "RemoveContainer" containerID="732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8" Oct 30 08:00:17.425373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.425137 5894 scope.go:122] "RemoveContainer" containerID="916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc" Oct 30 08:00:17.425373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.425191 5894 scope.go:122] "RemoveContainer" containerID="6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561" Oct 30 08:00:17.524169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.524124 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:00:17.669503 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.669409 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerStarted","Data":"ea03d4ac419300af31bd0d18eed56c3ce4baab004a4d7b7b4ab908130aceec86"} Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.675437 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerStarted","Data":"6fad7c920e91f3c2f7b745b6b1923c6406a580d33bdf39f5b6ae04fac13953c1"} Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.677688 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerStarted","Data":"d2ef010a954c476cc99a878accfbc045adc9627f3f1d5e57d67d89345158d883"} Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.680114 5894 generic.go:356] "Generic (PLEG): container finished" podID="eaac19a2-19af-431e-9553-4eaea056ed79" containerID="c44d32dddcc871bc1b91b350bf11dd610d9a09b4aac4945304101217f4041dd4" exitCode=60 Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.680198 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" event={"ID":"eaac19a2-19af-431e-9553-4eaea056ed79","Type":"ContainerDied","Data":"c44d32dddcc871bc1b91b350bf11dd610d9a09b4aac4945304101217f4041dd4"} Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.680269 5894 scope.go:122] "RemoveContainer" containerID="0a952b9bc1ca3bd0ada88bd90a7f5de7245b06d58e4ef2ca4ed6dd8b6dbd93cf" Oct 30 08:00:17.681082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.680800 5894 scope.go:122] "RemoveContainer" containerID="c44d32dddcc871bc1b91b350bf11dd610d9a09b4aac4945304101217f4041dd4" Oct 30 08:00:17.682171 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:17.681139 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz_odh-ai-gateway-infra(eaac19a2-19af-431e-9553-4eaea056ed79)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" Oct 30 08:00:17.682820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.682794 5894 generic.go:356] "Generic (PLEG): container finished" podID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerID="c096bbac74d48fef5e484587512cff95ecec59a2bef2f5d4e4a51c867b9fd486" exitCode=60 Oct 30 08:00:17.682948 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.682924 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerDied","Data":"c096bbac74d48fef5e484587512cff95ecec59a2bef2f5d4e4a51c867b9fd486"} Oct 30 08:00:17.683320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.683303 5894 scope.go:122] "RemoveContainer" containerID="c096bbac74d48fef5e484587512cff95ecec59a2bef2f5d4e4a51c867b9fd486" Oct 30 08:00:17.683620 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:17.683597 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9_odh-ai-gateway-infra(cfa7e21f-3373-4018-ae96-582ce2ff9f59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" Oct 30 08:00:17.684881 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.684859 5894 generic.go:356] "Generic (PLEG): container finished" podID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerID="cbc6f7016a75d882a9802ab8916ba86fcc7af7004f4a04bb1a92b0d8245b11c5" exitCode=60 Oct 30 08:00:17.684987 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.684931 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerDied","Data":"cbc6f7016a75d882a9802ab8916ba86fcc7af7004f4a04bb1a92b0d8245b11c5"} Oct 30 08:00:17.685257 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.685242 5894 scope.go:122] "RemoveContainer" containerID="cbc6f7016a75d882a9802ab8916ba86fcc7af7004f4a04bb1a92b0d8245b11c5" Oct 30 08:00:17.685502 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:17.685485 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp_odh-ai-gateway-infra(2feccda1-6ff1-4e55-b1ae-28e499ac0f42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" Oct 30 08:00:17.746537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.746473 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" podStartSLOduration=17.746459006 podStartE2EDuration="17.746459006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:17.742835693 +0000 UTC m=+3781.433753140" watchObservedRunningTime="2026-10-30 08:00:17.746459006 +0000 UTC m=+3781.437376442" Oct 30 08:00:17.787142 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.787085 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" podStartSLOduration=17.787070721 podStartE2EDuration="17.787070721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:17.785355946 +0000 UTC m=+3781.476273383" watchObservedRunningTime="2026-10-30 08:00:17.787070721 +0000 UTC m=+3781.477988159" Oct 30 08:00:17.928573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.928452 5894 scope.go:122] "RemoveContainer" containerID="0e312a7e3059fb9dc909e8a8c884d6a2f223caa624cc2d581f72b2a59942032e" Oct 30 08:00:17.957587 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.957554 5894 scope.go:122] "RemoveContainer" containerID="87e067988084809a995fbdae8b0b132ed1813b1addb56f48689621f2a72b00be" Oct 30 08:00:17.978367 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:17.978322 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:18.424281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.424244 5894 scope.go:122] "RemoveContainer" containerID="64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b" Oct 30 08:00:18.424468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.424423 5894 scope.go:122] "RemoveContainer" containerID="06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb" Oct 30 08:00:18.424570 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.424547 5894 scope.go:122] "RemoveContainer" containerID="99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c" Oct 30 08:00:18.424674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.424617 5894 scope.go:122] "RemoveContainer" containerID="063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202" Oct 30 08:00:18.424776 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.424757 5894 scope.go:122] "RemoveContainer" containerID="c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb" Oct 30 08:00:18.436848 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.436816 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" path="/var/lib/kubelet/pods/7423ac5b-14cf-4d34-a97a-f0156336a7c6/volumes" Oct 30 08:00:18.698604 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.698571 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerStarted","Data":"3fee5966b04c66e427ba27a78fa00d499737b04ba15128ce0bb13776ac43e270"} Oct 30 08:00:18.702695 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.702666 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerStarted","Data":"f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3"} Oct 30 08:00:18.703095 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.703079 5894 scope.go:122] "RemoveContainer" containerID="f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3" Oct 30 08:00:18.703448 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:18.703428 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh_odh-ai-gateway-infra(1c541641-0030-4fa0-8a8d-516668265dc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" Oct 30 08:00:18.704540 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.704520 5894 generic.go:356] "Generic (PLEG): container finished" podID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerID="ea03d4ac419300af31bd0d18eed56c3ce4baab004a4d7b7b4ab908130aceec86" exitCode=60 Oct 30 08:00:18.704678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.704589 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerDied","Data":"ea03d4ac419300af31bd0d18eed56c3ce4baab004a4d7b7b4ab908130aceec86"} Oct 30 08:00:18.704678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.704652 5894 scope.go:122] "RemoveContainer" containerID="ca94050972133b7b63aaa32090af222f3e9893818e07c67bc0635098de3dbff5" Oct 30 08:00:18.705004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.704986 5894 scope.go:122] "RemoveContainer" containerID="ea03d4ac419300af31bd0d18eed56c3ce4baab004a4d7b7b4ab908130aceec86" Oct 30 08:00:18.705342 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:18.705315 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n_odh-ai-gateway-infra(ef6b14ef-4cdd-4afd-9668-abfd139020d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" Oct 30 08:00:18.706671 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.706610 5894 generic.go:356] "Generic (PLEG): container finished" podID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerID="6fad7c920e91f3c2f7b745b6b1923c6406a580d33bdf39f5b6ae04fac13953c1" exitCode=60 Oct 30 08:00:18.706671 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.706659 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerDied","Data":"6fad7c920e91f3c2f7b745b6b1923c6406a580d33bdf39f5b6ae04fac13953c1"} Oct 30 08:00:18.707032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.707014 5894 scope.go:122] "RemoveContainer" containerID="6fad7c920e91f3c2f7b745b6b1923c6406a580d33bdf39f5b6ae04fac13953c1" Oct 30 08:00:18.707343 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:18.707321 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4_odh-ai-gateway-infra(09c415f0-2dcb-4da0-b3eb-f26abdb53126)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" Oct 30 08:00:18.708787 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.708767 5894 generic.go:356] "Generic (PLEG): container finished" podID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerID="d2ef010a954c476cc99a878accfbc045adc9627f3f1d5e57d67d89345158d883" exitCode=60 Oct 30 08:00:18.708890 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.708839 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerDied","Data":"d2ef010a954c476cc99a878accfbc045adc9627f3f1d5e57d67d89345158d883"} Oct 30 08:00:18.709200 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.709182 5894 scope.go:122] "RemoveContainer" containerID="d2ef010a954c476cc99a878accfbc045adc9627f3f1d5e57d67d89345158d883" Oct 30 08:00:18.709498 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:18.709479 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts_odh-ai-gateway-infra(3eced14c-1e20-4a1e-bc1b-bc705c0dc0de)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" Oct 30 08:00:18.712503 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.712482 5894 generic.go:356] "Generic (PLEG): container finished" podID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerID="f3e456e71953dd4f06bac1af3b2148535937f707cfd0471fc8de2935aaab394e" exitCode=60 Oct 30 08:00:18.712608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.712555 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" event={"ID":"1ec31c06-c07b-4d87-852a-cad68487b55b","Type":"ContainerDied","Data":"f3e456e71953dd4f06bac1af3b2148535937f707cfd0471fc8de2935aaab394e"} Oct 30 08:00:18.712938 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.712919 5894 scope.go:122] "RemoveContainer" containerID="f3e456e71953dd4f06bac1af3b2148535937f707cfd0471fc8de2935aaab394e" Oct 30 08:00:18.713238 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:18.713216 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl_odh-ai-gateway-infra(1ec31c06-c07b-4d87-852a-cad68487b55b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" Oct 30 08:00:18.744049 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.743997 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" podStartSLOduration=18.743984364 podStartE2EDuration="18.743984364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:18.739802296 +0000 UTC m=+3782.430719733" watchObservedRunningTime="2026-10-30 08:00:18.743984364 +0000 UTC m=+3782.434901779" Oct 30 08:00:18.756338 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.756308 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:18.799041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.799011 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:18.843337 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.843272 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:18.858976 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.858946 5894 scope.go:122] "RemoveContainer" containerID="f24c28f2a4bbf905b3c68cc49ed54ebecf4ec721f10e74e0f195bbceb8b138c4" Oct 30 08:00:18.880449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.880418 5894 scope.go:122] "RemoveContainer" containerID="1851860f21476e77a86a3826048ce257ae1d2dd952f267c488bfa0564274388d" Oct 30 08:00:18.883587 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.882413 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:18.902001 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.901975 5894 scope.go:122] "RemoveContainer" containerID="6f094289fa48b54fb2b7c255211a6b1dab0d03eb6256eff7be9947f2427cb561" Oct 30 08:00:18.921941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.921916 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:18.971737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:18.971652 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:19.139187 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.139159 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:19.157259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.157228 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\" (UniqueName: \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\") pod \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " Oct 30 08:00:19.157737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.157711 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\") pod \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\" (UID: \"2feccda1-6ff1-4e55-b1ae-28e499ac0f42\") " Oct 30 08:00:19.158726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.158702 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca" pod "2feccda1-6ff1-4e55-b1ae-28e499ac0f42" (UID: "2feccda1-6ff1-4e55-b1ae-28e499ac0f42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.177937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.176617 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks" pod "2feccda1-6ff1-4e55-b1ae-28e499ac0f42" (UID: "2feccda1-6ff1-4e55-b1ae-28e499ac0f42"). InnerVolumeSpecName "kube-api-access-jx7ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.266373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.266341 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.266866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.266846 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx7ks\" (UniqueName: \"kubernetes.io/projected/2feccda1-6ff1-4e55-b1ae-28e499ac0f42-kube-api-access-jx7ks\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.278238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.278210 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:19.310526 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.310496 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:19.385898 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.385860 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\") pod \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " Oct 30 08:00:19.386240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.386214 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\") pod \"eaac19a2-19af-431e-9553-4eaea056ed79\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " Oct 30 08:00:19.386379 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.386310 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\" (UniqueName: \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\") pod \"eaac19a2-19af-431e-9553-4eaea056ed79\" (UID: \"eaac19a2-19af-431e-9553-4eaea056ed79\") " Oct 30 08:00:19.386379 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.386355 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\" (UniqueName: \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\") pod \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\" (UID: \"cfa7e21f-3373-4018-ae96-582ce2ff9f59\") " Oct 30 08:00:19.386668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.386595 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca" pod "cfa7e21f-3373-4018-ae96-582ce2ff9f59" (UID: "cfa7e21f-3373-4018-ae96-582ce2ff9f59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.386835 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.386808 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca" pod "eaac19a2-19af-431e-9553-4eaea056ed79" (UID: "eaac19a2-19af-431e-9553-4eaea056ed79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:19.387106 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.387076 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7e21f-3373-4018-ae96-582ce2ff9f59-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.387206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.387108 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eaac19a2-19af-431e-9553-4eaea056ed79-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.389525 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.389501 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk" pod "cfa7e21f-3373-4018-ae96-582ce2ff9f59" (UID: "cfa7e21f-3373-4018-ae96-582ce2ff9f59"). InnerVolumeSpecName "kube-api-access-cwgrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.389629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.389539 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c" pod "eaac19a2-19af-431e-9553-4eaea056ed79" (UID: "eaac19a2-19af-431e-9553-4eaea056ed79"). InnerVolumeSpecName "kube-api-access-6qw4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:19.424356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.424320 5894 scope.go:122] "RemoveContainer" containerID="d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4" Oct 30 08:00:19.424569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.424551 5894 scope.go:122] "RemoveContainer" containerID="d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7" Oct 30 08:00:19.488793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.488752 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qw4c\" (UniqueName: \"kubernetes.io/projected/eaac19a2-19af-431e-9553-4eaea056ed79-kube-api-access-6qw4c\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.488793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.488783 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwgrk\" (UniqueName: \"kubernetes.io/projected/cfa7e21f-3373-4018-ae96-582ce2ff9f59-kube-api-access-cwgrk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:19.729316 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.729216 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerStarted","Data":"6afdfaa13a761ba415f47f55ccf346d78c88dc251e897bdc28183ac52ecb92a6"} Oct 30 08:00:19.744774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.744715 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerStarted","Data":"221f98688780bfc02bc2f77717c728808ef2c3b2f1928ecb0f945558ad558028"} Oct 30 08:00:19.747442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.747407 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerStarted","Data":"233203451d357de856ffb3a9ff334b7827b0b21261a39fa6f37f97cf7ebd3774"} Oct 30 08:00:19.749622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.749596 5894 generic.go:356] "Generic (PLEG): container finished" podID="01e99876-ea97-489e-9728-25fa00c726d6" containerID="3fee5966b04c66e427ba27a78fa00d499737b04ba15128ce0bb13776ac43e270" exitCode=60 Oct 30 08:00:19.749800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.749680 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerDied","Data":"3fee5966b04c66e427ba27a78fa00d499737b04ba15128ce0bb13776ac43e270"} Oct 30 08:00:19.749800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.749729 5894 scope.go:122] "RemoveContainer" containerID="916e93b6a335b84f1550bdacef656531b726540db5e7793ce141a7081e9aeabc" Oct 30 08:00:19.750073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.750058 5894 scope.go:122] "RemoveContainer" containerID="3fee5966b04c66e427ba27a78fa00d499737b04ba15128ce0bb13776ac43e270" Oct 30 08:00:19.750400 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:19.750372 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm_odh-ai-gateway-infra(01e99876-ea97-489e-9728-25fa00c726d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" podUID="01e99876-ea97-489e-9728-25fa00c726d6" Oct 30 08:00:19.751429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.751408 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" Oct 30 08:00:19.751549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.751422 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp" event={"ID":"2feccda1-6ff1-4e55-b1ae-28e499ac0f42","Type":"ContainerDied","Data":"e21e722406ea728106a4931d274e24a10ae42e23b6fdc96a6939c0b5240923f1"} Oct 30 08:00:19.752751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.752728 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:19.753723 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.753702 5894 generic.go:356] "Generic (PLEG): container finished" podID="1c541641-0030-4fa0-8a8d-516668265dc3" containerID="f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3" exitCode=60 Oct 30 08:00:19.753840 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.753780 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerDied","Data":"f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3"} Oct 30 08:00:19.754185 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.754160 5894 scope.go:122] "RemoveContainer" containerID="f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3" Oct 30 08:00:19.754465 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:19.754442 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh_odh-ai-gateway-infra(1c541641-0030-4fa0-8a8d-516668265dc3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" Oct 30 08:00:19.760381 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.760353 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerStarted","Data":"fadc58e656ac18cf2e054db68615716158cb32ee3c6f4b949e80cffdfdd1c80c"} Oct 30 08:00:19.763175 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.763148 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerStarted","Data":"14fbce3cfc7094d2fab365c12c09554ebad85d1f06f6870be988e201ac96aaac"} Oct 30 08:00:19.770420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.770394 5894 scope.go:122] "RemoveContainer" containerID="cbc6f7016a75d882a9802ab8916ba86fcc7af7004f4a04bb1a92b0d8245b11c5" Oct 30 08:00:19.783214 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.783186 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" event={"ID":"eaac19a2-19af-431e-9553-4eaea056ed79","Type":"ContainerDied","Data":"f86e2de77b9b6897dadd1d3d579c5cc3c314a66dcdb67572cddc6b0d4ad703cc"} Oct 30 08:00:19.783465 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.783449 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz" Oct 30 08:00:19.784550 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.784493 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" podStartSLOduration=19.784472995 podStartE2EDuration="19.784472995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:19.780912748 +0000 UTC m=+3783.471830187" watchObservedRunningTime="2026-10-30 08:00:19.784472995 +0000 UTC m=+3783.475390432" Oct 30 08:00:19.793982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.793946 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" event={"ID":"cfa7e21f-3373-4018-ae96-582ce2ff9f59","Type":"ContainerDied","Data":"b4fc934a44c0d6f70bec93a37a33f70948881a3cfc3c5d3fd5a634ab5e24c7b0"} Oct 30 08:00:19.794299 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.794280 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9" Oct 30 08:00:19.863674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.863165 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:19.958727 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.958151 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:19.995135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:19.995091 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:20.100157 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.100124 5894 scope.go:122] "RemoveContainer" containerID="732e0c149f0fa3f3e904bca2319b15e0bca48fe9bb33cffe46ecc6156cc76ef8" Oct 30 08:00:20.110811 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.110647 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" podStartSLOduration=20.110607177 podStartE2EDuration="20.110607177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:20.107462513 +0000 UTC m=+3783.798379953" watchObservedRunningTime="2026-10-30 08:00:20.110607177 +0000 UTC m=+3783.801524613" Oct 30 08:00:20.152609 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.152574 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:20.162838 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.162805 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e5fa2d-29889120-pl2sp"] Oct 30 08:00:20.171536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.171485 5894 scope.go:122] "RemoveContainer" containerID="c44d32dddcc871bc1b91b350bf11dd610d9a09b4aac4945304101217f4041dd4" Oct 30 08:00:20.179026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.178735 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:20.184173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.184150 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:20.189578 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.189508 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:20.189720 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.189602 5894 scope.go:122] "RemoveContainer" containerID="c096bbac74d48fef5e484587512cff95ecec59a2bef2f5d4e4a51c867b9fd486" Oct 30 08:00:20.219219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.219192 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:20.228314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.228281 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx"] Oct 30 08:00:20.229147 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229123 5894 cpu_manager.go:414] "containerMap: removing container" podUID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" containerName="payload-processing" Oct 30 08:00:20.229147 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229147 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" containerName="payload-processing" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229165 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8c9ef108-46fa-458b-af70-90065ca8c203" containerName="revoke-keys" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229173 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9ef108-46fa-458b-af70-90065ca8c203" containerName="revoke-keys" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229203 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229214 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229228 5894 cpu_manager.go:414] "containerMap: removing container" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229236 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229245 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229251 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229267 5894 cpu_manager.go:414] "containerMap: removing container" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229273 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229289 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229297 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229314 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229321 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229336 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.229339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229343 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229353 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229361 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229374 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229382 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229394 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229401 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229415 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229423 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229442 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229449 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229462 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229469 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229480 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229487 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229498 5894 cpu_manager.go:414] "containerMap: removing container" podUID="adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" containerName="revoke-keys" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229505 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" containerName="revoke-keys" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229520 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229526 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229551 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229558 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229568 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229575 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229803 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229817 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229827 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229837 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229848 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229859 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229868 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229876 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229886 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229899 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229910 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229919 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229930 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229940 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" containerName="revoke-keys" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229953 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229966 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff106477-2682-4035-bfda-adffc03ed0c7" containerName="istio-proxy" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229975 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229984 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.229994 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.230006 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="7423ac5b-14cf-4d34-a97a-f0156336a7c6" containerName="payload-processing" Oct 30 08:00:20.230219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.230015 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9ef108-46fa-458b-af70-90065ca8c203" containerName="revoke-keys" Oct 30 08:00:20.234823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.234802 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.240588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240558 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\") pod \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " Oct 30 08:00:20.240733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240654 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\") pod \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " Oct 30 08:00:20.240788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240766 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\") pod \"1ec31c06-c07b-4d87-852a-cad68487b55b\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " Oct 30 08:00:20.240873 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240851 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\" (UniqueName: \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\") pod \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\" (UID: \"09c415f0-2dcb-4da0-b3eb-f26abdb53126\") " Oct 30 08:00:20.240927 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240910 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca" pod "09c415f0-2dcb-4da0-b3eb-f26abdb53126" (UID: "09c415f0-2dcb-4da0-b3eb-f26abdb53126"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.240927 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240921 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\" (UniqueName: \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\") pod \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\" (UID: \"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de\") " Oct 30 08:00:20.241032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.240954 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\" (UniqueName: \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\") pod \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " Oct 30 08:00:20.241032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241025 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\") pod \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\" (UID: \"ef6b14ef-4cdd-4afd-9668-abfd139020d1\") " Oct 30 08:00:20.241193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241034 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca" pod "3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" (UID: "3eced14c-1e20-4a1e-bc1b-bc705c0dc0de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.241193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241099 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\" (UniqueName: \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\") pod \"1ec31c06-c07b-4d87-852a-cad68487b55b\" (UID: \"1ec31c06-c07b-4d87-852a-cad68487b55b\") " Oct 30 08:00:20.241193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241129 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca" pod "1ec31c06-c07b-4d87-852a-cad68487b55b" (UID: "1ec31c06-c07b-4d87-852a-cad68487b55b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.241483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241458 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca" pod "ef6b14ef-4cdd-4afd-9668-abfd139020d1" (UID: "ef6b14ef-4cdd-4afd-9668-abfd139020d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:20.241483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241467 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.241642 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241609 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.241754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241664 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmgz\" (UniqueName: \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.241973 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241955 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09c415f0-2dcb-4da0-b3eb-f26abdb53126-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.242047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241979 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.242047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.241993 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ec31c06-c07b-4d87-852a-cad68487b55b-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.242047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.242008 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef6b14ef-4cdd-4afd-9668-abfd139020d1-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.243677 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.243607 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4" pod "09c415f0-2dcb-4da0-b3eb-f26abdb53126" (UID: "09c415f0-2dcb-4da0-b3eb-f26abdb53126"). InnerVolumeSpecName "kube-api-access-476k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.244027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.244007 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n" pod "3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" (UID: "3eced14c-1e20-4a1e-bc1b-bc705c0dc0de"). InnerVolumeSpecName "kube-api-access-df24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.244207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.244185 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66" pod "ef6b14ef-4cdd-4afd-9668-abfd139020d1" (UID: "ef6b14ef-4cdd-4afd-9668-abfd139020d1"). InnerVolumeSpecName "kube-api-access-h4n66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.244274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.244234 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq" pod "1ec31c06-c07b-4d87-852a-cad68487b55b" (UID: "1ec31c06-c07b-4d87-852a-cad68487b55b"). InnerVolumeSpecName "kube-api-access-lbtcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:20.259857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.259824 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:00:20.263484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.263446 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx"] Oct 30 08:00:20.299355 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.299291 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" podStartSLOduration=20.299272773 podStartE2EDuration="20.299272773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:20.297825296 +0000 UTC m=+3783.988742733" watchObservedRunningTime="2026-10-30 08:00:20.299272773 +0000 UTC m=+3783.990190215" Oct 30 08:00:20.343355 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343317 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.343541 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343433 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.343541 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343463 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmgz\" (UniqueName: \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.343541 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343532 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtcq\" (UniqueName: \"kubernetes.io/projected/1ec31c06-c07b-4d87-852a-cad68487b55b-kube-api-access-lbtcq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.343541 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343544 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-476k4\" (UniqueName: \"kubernetes.io/projected/09c415f0-2dcb-4da0-b3eb-f26abdb53126-kube-api-access-476k4\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.343870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343555 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df24n\" (UniqueName: \"kubernetes.io/projected/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de-kube-api-access-df24n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.343870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.343565 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4n66\" (UniqueName: \"kubernetes.io/projected/ef6b14ef-4cdd-4afd-9668-abfd139020d1-kube-api-access-h4n66\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:20.344253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.344230 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.345783 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.345762 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.369488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.369459 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmgz\" (UniqueName: \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\") pod \"maas-api-revoke-sub-aa23cd32-s8brx\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.439670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.439598 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" path="/var/lib/kubelet/pods/2feccda1-6ff1-4e55-b1ae-28e499ac0f42/volumes" Oct 30 08:00:20.552860 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.552830 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:20.557373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.557154 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:20.568360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.568328 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-228571-29889120-szsj9"] Oct 30 08:00:20.615939 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.612889 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:20.618845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.618686 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-54f2b2-29889120-hwmtz"] Oct 30 08:00:20.792378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.792344 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:20.803991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.803891 5894 generic.go:356] "Generic (PLEG): container finished" podID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerID="fadc58e656ac18cf2e054db68615716158cb32ee3c6f4b949e80cffdfdd1c80c" exitCode=60 Oct 30 08:00:20.803991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.803972 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerDied","Data":"fadc58e656ac18cf2e054db68615716158cb32ee3c6f4b949e80cffdfdd1c80c"} Oct 30 08:00:20.804231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.804048 5894 scope.go:122] "RemoveContainer" containerID="64ef834f9424df3877cb818271a929982c94d8c548c9f77029fb019678145f7b" Oct 30 08:00:20.804509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.804490 5894 scope.go:122] "RemoveContainer" containerID="fadc58e656ac18cf2e054db68615716158cb32ee3c6f4b949e80cffdfdd1c80c" Oct 30 08:00:20.804940 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:20.804907 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n_odh-ai-gateway-infra(5c3c9121-b3c1-45e1-97d8-393910981f85)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" Oct 30 08:00:20.806588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.806564 5894 generic.go:356] "Generic (PLEG): container finished" podID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerID="14fbce3cfc7094d2fab365c12c09554ebad85d1f06f6870be988e201ac96aaac" exitCode=60 Oct 30 08:00:20.806716 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.806665 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerDied","Data":"14fbce3cfc7094d2fab365c12c09554ebad85d1f06f6870be988e201ac96aaac"} Oct 30 08:00:20.807108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.807087 5894 scope.go:122] "RemoveContainer" containerID="14fbce3cfc7094d2fab365c12c09554ebad85d1f06f6870be988e201ac96aaac" Oct 30 08:00:20.807532 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:20.807499 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g_odh-ai-gateway-infra(9a600d24-d431-4166-a41c-4f3c3012d5ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" Oct 30 08:00:20.808708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.808687 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" Oct 30 08:00:20.808826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.808716 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4" event={"ID":"09c415f0-2dcb-4da0-b3eb-f26abdb53126","Type":"ContainerDied","Data":"9283715216bf6bc0a1d750d804e774d37a4ae414c77feb1ac75896b346e42048"} Oct 30 08:00:20.810356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.810337 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" Oct 30 08:00:20.810460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.810361 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts" event={"ID":"3eced14c-1e20-4a1e-bc1b-bc705c0dc0de","Type":"ContainerDied","Data":"336dee75427daf3ef26bef80bc27b8c9e38f54a436df8bc209fa818e58380b0f"} Oct 30 08:00:20.813423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.813397 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" Oct 30 08:00:20.813537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.813404 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl" event={"ID":"1ec31c06-c07b-4d87-852a-cad68487b55b","Type":"ContainerDied","Data":"1b18898b4600a9a2ae548ac1849448741ab826c92593da5b10f8ac438464ee07"} Oct 30 08:00:20.817155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.817120 5894 generic.go:356] "Generic (PLEG): container finished" podID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerID="6afdfaa13a761ba415f47f55ccf346d78c88dc251e897bdc28183ac52ecb92a6" exitCode=60 Oct 30 08:00:20.817263 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.817204 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerDied","Data":"6afdfaa13a761ba415f47f55ccf346d78c88dc251e897bdc28183ac52ecb92a6"} Oct 30 08:00:20.820129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.820104 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerStarted","Data":"c83408c2f63456c8b027e47ccf98b59c5d35e047e5ef26210174a9933c9217de"} Oct 30 08:00:20.822206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.822188 5894 scope.go:122] "RemoveContainer" containerID="063ec5f47e797a9d113dd1725c8c4105247b2169d7ffc0f4c63faf4f40b5f202" Oct 30 08:00:20.822460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.822438 5894 generic.go:356] "Generic (PLEG): container finished" podID="fa98e977-1e85-4da0-92e2-55e96732b578" containerID="221f98688780bfc02bc2f77717c728808ef2c3b2f1928ecb0f945558ad558028" exitCode=60 Oct 30 08:00:20.822528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.822501 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerDied","Data":"221f98688780bfc02bc2f77717c728808ef2c3b2f1928ecb0f945558ad558028"} Oct 30 08:00:20.822987 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.822968 5894 scope.go:122] "RemoveContainer" containerID="221f98688780bfc02bc2f77717c728808ef2c3b2f1928ecb0f945558ad558028" Oct 30 08:00:20.823335 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:20.823312 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5_odh-ai-gateway-infra(fa98e977-1e85-4da0-92e2-55e96732b578)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" Oct 30 08:00:20.824985 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.824961 5894 generic.go:356] "Generic (PLEG): container finished" podID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerID="233203451d357de856ffb3a9ff334b7827b0b21261a39fa6f37f97cf7ebd3774" exitCode=60 Oct 30 08:00:20.825102 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.824980 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerDied","Data":"233203451d357de856ffb3a9ff334b7827b0b21261a39fa6f37f97cf7ebd3774"} Oct 30 08:00:20.825484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.825462 5894 scope.go:122] "RemoveContainer" containerID="233203451d357de856ffb3a9ff334b7827b0b21261a39fa6f37f97cf7ebd3774" Oct 30 08:00:20.825827 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:20.825804 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9_odh-ai-gateway-infra(9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" Oct 30 08:00:20.830381 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.830348 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerStarted","Data":"5c6646c5f6d373d005c84a478bda227cd0568a152c2bc9d4afc7f388641b3df4"} Oct 30 08:00:20.834794 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.834285 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" event={"ID":"ef6b14ef-4cdd-4afd-9668-abfd139020d1","Type":"ContainerDied","Data":"83dde05ec8a4b176f7437265bc045d68a06ac9a41a3611d15e235587f589bb4d"} Oct 30 08:00:20.834794 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.834319 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n" Oct 30 08:00:20.841411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.841377 5894 scope.go:122] "RemoveContainer" containerID="6fad7c920e91f3c2f7b745b6b1923c6406a580d33bdf39f5b6ae04fac13953c1" Oct 30 08:00:20.876210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.875542 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx"] Oct 30 08:00:20.937129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.937094 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:20.968786 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:20.968756 5894 scope.go:122] "RemoveContainer" containerID="d2ef010a954c476cc99a878accfbc045adc9627f3f1d5e57d67d89345158d883" Oct 30 08:00:21.084549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.084463 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:21.133185 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.133145 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz"] Oct 30 08:00:21.158570 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.158542 5894 scope.go:122] "RemoveContainer" containerID="f3e456e71953dd4f06bac1af3b2148535937f707cfd0471fc8de2935aaab394e" Oct 30 08:00:21.159977 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.159953 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41aef6d6-pkqmz"] Oct 30 08:00:21.164042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.164014 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:21.172418 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.172392 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:21.173513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.173495 5894 scope.go:122] "RemoveContainer" containerID="06d3143a73a425d2e710f7d605e1329dca629d2344e371931f2947d34dc619eb" Oct 30 08:00:21.176523 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.176502 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:21.180569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.180547 5894 scope.go:122] "RemoveContainer" containerID="fea161c70814f11856a7bb1fbb628316693e6acc703a83d0f5fb894268c96f32" Oct 30 08:00:21.181754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.181554 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:21.191841 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.191812 5894 scope.go:122] "RemoveContainer" containerID="99330f06e152aace976caf54f40de734d7c425bc7b73790265c969269f513a5c" Oct 30 08:00:21.192277 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.192257 5894 scope.go:122] "RemoveContainer" containerID="85349d363f5f144cf526eead54cd5a2f0701bab36bbdc8736617fe0312a3b1e1" Oct 30 08:00:21.207832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.207808 5894 scope.go:122] "RemoveContainer" containerID="64e5508623f18a9587a86d42694dbbc9b52c6ce2b2ee8c3416ac8dcb38466bcc" Oct 30 08:00:21.208218 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.208199 5894 scope.go:122] "RemoveContainer" containerID="c3e33dab75fe70d6b753cfeddca2f69b1ec3efa65ea47e0271774a0776d70acb" Oct 30 08:00:21.217362 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.217335 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:21.223984 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.223957 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1e0880-29889120-njxq4"] Oct 30 08:00:21.227242 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.227223 5894 scope.go:122] "RemoveContainer" containerID="ea03d4ac419300af31bd0d18eed56c3ce4baab004a4d7b7b4ab908130aceec86" Oct 30 08:00:21.263606 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.263570 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:21.268628 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.268566 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\") pod \"1c541641-0030-4fa0-8a8d-516668265dc3\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " Oct 30 08:00:21.268795 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.268776 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\") pod \"01e99876-ea97-489e-9728-25fa00c726d6\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " Oct 30 08:00:21.268888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.268832 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\" (UniqueName: \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\") pod \"01e99876-ea97-489e-9728-25fa00c726d6\" (UID: \"01e99876-ea97-489e-9728-25fa00c726d6\") " Oct 30 08:00:21.268888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.268879 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\" (UniqueName: \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\") pod \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " Oct 30 08:00:21.269077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269022 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\" (UniqueName: \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\") pod \"1c541641-0030-4fa0-8a8d-516668265dc3\" (UID: \"1c541641-0030-4fa0-8a8d-516668265dc3\") " Oct 30 08:00:21.269077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269044 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca" pod "1c541641-0030-4fa0-8a8d-516668265dc3" (UID: "1c541641-0030-4fa0-8a8d-516668265dc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:21.269077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269062 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\") pod \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\" (UID: \"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea\") " Oct 30 08:00:21.269505 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269479 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca" pod "abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" (UID: "abbe9d7a-6ea5-4ca5-8d46-863cec9cedea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:21.269677 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269552 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca" pod "01e99876-ea97-489e-9728-25fa00c726d6" (UID: "01e99876-ea97-489e-9728-25fa00c726d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:21.269960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269936 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c541641-0030-4fa0-8a8d-516668265dc3-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.270079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269963 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01e99876-ea97-489e-9728-25fa00c726d6-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.270079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.269979 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.271354 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.271331 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-bd38cf-29889120-vnj2n"] Oct 30 08:00:21.272716 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.272670 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c" pod "01e99876-ea97-489e-9728-25fa00c726d6" (UID: "01e99876-ea97-489e-9728-25fa00c726d6"). InnerVolumeSpecName "kube-api-access-6tl2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:21.272853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.272832 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd" pod "1c541641-0030-4fa0-8a8d-516668265dc3" (UID: "1c541641-0030-4fa0-8a8d-516668265dc3"). InnerVolumeSpecName "kube-api-access-96hpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:21.274361 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.274334 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk" pod "abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" (UID: "abbe9d7a-6ea5-4ca5-8d46-863cec9cedea"). InnerVolumeSpecName "kube-api-access-hhqtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:21.317135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.317108 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:21.371519 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.371441 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96hpd\" (UniqueName: \"kubernetes.io/projected/1c541641-0030-4fa0-8a8d-516668265dc3-kube-api-access-96hpd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.371519 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.371471 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tl2c\" (UniqueName: \"kubernetes.io/projected/01e99876-ea97-489e-9728-25fa00c726d6-kube-api-access-6tl2c\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.371519 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.371484 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhqtk\" (UniqueName: \"kubernetes.io/projected/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea-kube-api-access-hhqtk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:21.376767 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.376742 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:21.386033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.386006 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-268576-29889120-kqxhl"] Oct 30 08:00:21.509669 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.509599 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:21.524377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.524336 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0f76b0-29889120-w86ts"] Oct 30 08:00:21.848009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.847985 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" Oct 30 08:00:21.848507 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.848021 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm" event={"ID":"01e99876-ea97-489e-9728-25fa00c726d6","Type":"ContainerDied","Data":"17abc25184e46d446f2d35f37714e25d085eeed3600f3b44fd94ef3387eebc41"} Oct 30 08:00:21.848507 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.848081 5894 scope.go:122] "RemoveContainer" containerID="3fee5966b04c66e427ba27a78fa00d499737b04ba15128ce0bb13776ac43e270" Oct 30 08:00:21.850531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.850503 5894 generic.go:356] "Generic (PLEG): container finished" podID="711294e8-5614-4075-8ab9-5898ca529105" containerID="5c6646c5f6d373d005c84a478bda227cd0568a152c2bc9d4afc7f388641b3df4" exitCode=60 Oct 30 08:00:21.850707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.850566 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerDied","Data":"5c6646c5f6d373d005c84a478bda227cd0568a152c2bc9d4afc7f388641b3df4"} Oct 30 08:00:21.851105 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.851078 5894 scope.go:122] "RemoveContainer" containerID="5c6646c5f6d373d005c84a478bda227cd0568a152c2bc9d4afc7f388641b3df4" Oct 30 08:00:21.851463 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:21.851438 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl_odh-ai-gateway-infra(711294e8-5614-4075-8ab9-5898ca529105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" podUID="711294e8-5614-4075-8ab9-5898ca529105" Oct 30 08:00:21.853077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.852767 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" event={"ID":"9e445752-1cf8-43cc-b495-61da204d5ee6","Type":"ContainerStarted","Data":"2d3f6f6a3eacb43eaa6b6959069db45e9e386541eb3f51e5203919db839c3758"} Oct 30 08:00:21.853077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.852816 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" event={"ID":"9e445752-1cf8-43cc-b495-61da204d5ee6","Type":"ContainerStarted","Data":"67b3cde7dafde9c453608dc4f78db81fcbfc6bd273cc022d2d52e8b559983f0b"} Oct 30 08:00:21.854461 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.854433 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" Oct 30 08:00:21.854461 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.854445 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh" event={"ID":"1c541641-0030-4fa0-8a8d-516668265dc3","Type":"ContainerDied","Data":"731f0bd4dcdbb5d01a810425cd33579fac6da08ce7b9cd5537bf236fe4f0de26"} Oct 30 08:00:21.864014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.863991 5894 scope.go:122] "RemoveContainer" containerID="d982eb90d22b61bfa7ef71ec2b3b13e40853feceae652ed51c4f670ee4efbef7" Oct 30 08:00:21.869561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.869515 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" event={"ID":"abbe9d7a-6ea5-4ca5-8d46-863cec9cedea","Type":"ContainerDied","Data":"43d55881be58e1fae29c30712913360c4258ebfca0be302edc3cafba2f11ae16"} Oct 30 08:00:21.870733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.870706 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254" Oct 30 08:00:21.875387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.875356 5894 generic.go:356] "Generic (PLEG): container finished" podID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerID="c83408c2f63456c8b027e47ccf98b59c5d35e047e5ef26210174a9933c9217de" exitCode=60 Oct 30 08:00:21.875568 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.875548 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerDied","Data":"c83408c2f63456c8b027e47ccf98b59c5d35e047e5ef26210174a9933c9217de"} Oct 30 08:00:21.876195 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.876174 5894 scope.go:122] "RemoveContainer" containerID="c83408c2f63456c8b027e47ccf98b59c5d35e047e5ef26210174a9933c9217de" Oct 30 08:00:21.876843 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:21.876811 5894 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj_odh-ai-gateway-infra(9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" Oct 30 08:00:21.917844 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.917810 5894 scope.go:122] "RemoveContainer" containerID="f46e4ddd353afda44d6c97d30964117720c5c805406fdb7dfb1ce003ff1febd3" Oct 30 08:00:21.923454 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:21.923186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:22.011222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.010862 5894 scope.go:122] "RemoveContainer" containerID="6afdfaa13a761ba415f47f55ccf346d78c88dc251e897bdc28183ac52ecb92a6" Oct 30 08:00:22.016558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.013921 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" podStartSLOduration=2.01390255 podStartE2EDuration="2.01390255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:21.981461397 +0000 UTC m=+3785.672378834" watchObservedRunningTime="2026-10-30 08:00:22.01390255 +0000 UTC m=+3785.704819988" Oct 30 08:00:22.018264 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.018234 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:22.036843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.036803 5894 scope.go:122] "RemoveContainer" containerID="d8afbf53a7dfae160571a84fb0640c176274adaa7e0212b4d168be08e786c3c4" Oct 30 08:00:22.094701 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.089874 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:22.119589 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.119555 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:22.195868 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.195836 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\") pod \"fa98e977-1e85-4da0-92e2-55e96732b578\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " Oct 30 08:00:22.196035 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.196023 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\" (UniqueName: \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\") pod \"fa98e977-1e85-4da0-92e2-55e96732b578\" (UID: \"fa98e977-1e85-4da0-92e2-55e96732b578\") " Oct 30 08:00:22.196290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.196265 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca" pod "fa98e977-1e85-4da0-92e2-55e96732b578" (UID: "fa98e977-1e85-4da0-92e2-55e96732b578"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:22.196616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.196596 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa98e977-1e85-4da0-92e2-55e96732b578-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.198181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.198159 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc" pod "fa98e977-1e85-4da0-92e2-55e96732b578" (UID: "fa98e977-1e85-4da0-92e2-55e96732b578"). InnerVolumeSpecName "kube-api-access-rqnmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:22.276167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.276136 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:22.280434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.280403 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:22.283860 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.283836 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:22.297524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297493 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\" (UniqueName: \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\") pod \"9a600d24-d431-4166-a41c-4f3c3012d5ad\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " Oct 30 08:00:22.297733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297575 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\") pod \"9a600d24-d431-4166-a41c-4f3c3012d5ad\" (UID: \"9a600d24-d431-4166-a41c-4f3c3012d5ad\") " Oct 30 08:00:22.297733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297613 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\" (UniqueName: \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\") pod \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " Oct 30 08:00:22.297733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297653 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\") pod \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\" (UID: \"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5\") " Oct 30 08:00:22.297733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297686 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\") pod \"5c3c9121-b3c1-45e1-97d8-393910981f85\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " Oct 30 08:00:22.298075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.297809 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\" (UniqueName: \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\") pod \"5c3c9121-b3c1-45e1-97d8-393910981f85\" (UID: \"5c3c9121-b3c1-45e1-97d8-393910981f85\") " Oct 30 08:00:22.298075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298007 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca" pod "9a600d24-d431-4166-a41c-4f3c3012d5ad" (UID: "9a600d24-d431-4166-a41c-4f3c3012d5ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:22.298075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298041 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca" pod "9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" (UID: "9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:22.298075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298047 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca" pod "5c3c9121-b3c1-45e1-97d8-393910981f85" (UID: "5c3c9121-b3c1-45e1-97d8-393910981f85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:22.298718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298693 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a600d24-d431-4166-a41c-4f3c3012d5ad-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.298718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298722 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.298902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298739 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqnmc\" (UniqueName: \"kubernetes.io/projected/fa98e977-1e85-4da0-92e2-55e96732b578-kube-api-access-rqnmc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.298902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.298754 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c3c9121-b3c1-45e1-97d8-393910981f85-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.300183 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.300151 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7" pod "9a600d24-d431-4166-a41c-4f3c3012d5ad" (UID: "9a600d24-d431-4166-a41c-4f3c3012d5ad"). InnerVolumeSpecName "kube-api-access-j6nl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:22.300311 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.300292 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh" pod "5c3c9121-b3c1-45e1-97d8-393910981f85" (UID: "5c3c9121-b3c1-45e1-97d8-393910981f85"). InnerVolumeSpecName "kube-api-access-fhlhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:22.300372 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.300359 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw" pod "9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" (UID: "9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5"). InnerVolumeSpecName "kube-api-access-szkxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:22.401270 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.401173 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6nl7\" (UniqueName: \"kubernetes.io/projected/9a600d24-d431-4166-a41c-4f3c3012d5ad-kube-api-access-j6nl7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.401270 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.401215 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkxw\" (UniqueName: \"kubernetes.io/projected/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5-kube-api-access-szkxw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.401270 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.401231 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhlhh\" (UniqueName: \"kubernetes.io/projected/5c3c9121-b3c1-45e1-97d8-393910981f85-kube-api-access-fhlhh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:22.409043 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.408983 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:22.414294 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.414261 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-705337-29889120-lpqvm"] Oct 30 08:00:22.442116 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.441562 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e99876-ea97-489e-9728-25fa00c726d6" path="/var/lib/kubelet/pods/01e99876-ea97-489e-9728-25fa00c726d6/volumes" Oct 30 08:00:22.444616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.444572 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" path="/var/lib/kubelet/pods/09c415f0-2dcb-4da0-b3eb-f26abdb53126/volumes" Oct 30 08:00:22.445522 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.445473 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" path="/var/lib/kubelet/pods/1ec31c06-c07b-4d87-852a-cad68487b55b/volumes" Oct 30 08:00:22.446320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.446298 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" path="/var/lib/kubelet/pods/3eced14c-1e20-4a1e-bc1b-bc705c0dc0de/volumes" Oct 30 08:00:22.452341 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.452317 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="424488d6-cb5f-472d-b53d-978f5cc918fa" path="/var/lib/kubelet/pods/424488d6-cb5f-472d-b53d-978f5cc918fa/volumes" Oct 30 08:00:22.453568 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.453547 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" path="/var/lib/kubelet/pods/cfa7e21f-3373-4018-ae96-582ce2ff9f59/volumes" Oct 30 08:00:22.454835 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.454799 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" path="/var/lib/kubelet/pods/eaac19a2-19af-431e-9553-4eaea056ed79/volumes" Oct 30 08:00:22.460490 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.460451 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6b14ef-4cdd-4afd-9668-abfd139020d1" path="/var/lib/kubelet/pods/ef6b14ef-4cdd-4afd-9668-abfd139020d1/volumes" Oct 30 08:00:22.500845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.500809 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:22.511083 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.511051 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-352fb3-29889120-m5mwh"] Oct 30 08:00:22.549395 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.549352 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:22.555827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.555789 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6051b2-29889120-ch254"] Oct 30 08:00:22.891853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.891826 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" Oct 30 08:00:22.892296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.891842 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5" event={"ID":"fa98e977-1e85-4da0-92e2-55e96732b578","Type":"ContainerDied","Data":"2ff3d5ed4788f09835a1fe55909118644aa8fda40e10a03c910fa77d91564f9e"} Oct 30 08:00:22.892296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.891913 5894 scope.go:122] "RemoveContainer" containerID="221f98688780bfc02bc2f77717c728808ef2c3b2f1928ecb0f945558ad558028" Oct 30 08:00:22.893547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.893477 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" event={"ID":"9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5","Type":"ContainerDied","Data":"9016f9683ae5b082eb88ddb3cec102a54ec6f8790230528902b90d383d16ba8e"} Oct 30 08:00:22.893547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.893534 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9" Oct 30 08:00:22.901071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.901014 5894 generic.go:356] "Generic (PLEG): container finished" podID="9e445752-1cf8-43cc-b495-61da204d5ee6" containerID="2d3f6f6a3eacb43eaa6b6959069db45e9e386541eb3f51e5203919db839c3758" exitCode=0 Oct 30 08:00:22.901181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.901166 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" event={"ID":"9e445752-1cf8-43cc-b495-61da204d5ee6","Type":"ContainerDied","Data":"2d3f6f6a3eacb43eaa6b6959069db45e9e386541eb3f51e5203919db839c3758"} Oct 30 08:00:22.904065 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.904022 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" event={"ID":"5c3c9121-b3c1-45e1-97d8-393910981f85","Type":"ContainerDied","Data":"5b9bed8afbadb6ff690442259e4f911d4f1c2956119cb860b54915fa6bcd26a8"} Oct 30 08:00:22.904172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.904062 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n" Oct 30 08:00:22.906531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.906503 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" event={"ID":"9a600d24-d431-4166-a41c-4f3c3012d5ad","Type":"ContainerDied","Data":"bce1930f775d52fdd3abf78a21bd0d0f39052875e6f2a0a2de3ca40d2a685ccb"} Oct 30 08:00:22.906685 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.906564 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g" Oct 30 08:00:22.921996 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.918793 5894 scope.go:122] "RemoveContainer" containerID="233203451d357de856ffb3a9ff334b7827b0b21261a39fa6f37f97cf7ebd3774" Oct 30 08:00:22.945693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.945671 5894 scope.go:122] "RemoveContainer" containerID="fadc58e656ac18cf2e054db68615716158cb32ee3c6f4b949e80cffdfdd1c80c" Oct 30 08:00:22.961671 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.961623 5894 scope.go:122] "RemoveContainer" containerID="14fbce3cfc7094d2fab365c12c09554ebad85d1f06f6870be988e201ac96aaac" Oct 30 08:00:22.974809 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.974744 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:22.980062 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:22.980022 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2f845b-29889120-9fr6n"] Oct 30 08:00:23.055656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.054608 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:23.065121 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.065073 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-4eff1f-29889120-4pqr9"] Oct 30 08:00:23.108888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.108335 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:23.108888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.108423 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e193ee-29889120-hns2g"] Oct 30 08:00:23.135744 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.135586 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:23.160055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.159882 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:23.183180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.183152 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:23.193764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.193729 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:23.197665 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.197599 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b72d79-29889120-clxr5"] Oct 30 08:00:23.213557 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.213523 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\" (UniqueName: \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\") pod \"711294e8-5614-4075-8ab9-5898ca529105\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " Oct 30 08:00:23.213750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.213707 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\") pod \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " Oct 30 08:00:23.213845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.213759 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\" (UniqueName: \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\") pod \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\" (UID: \"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef\") " Oct 30 08:00:23.213933 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.213911 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\") pod \"711294e8-5614-4075-8ab9-5898ca529105\" (UID: \"711294e8-5614-4075-8ab9-5898ca529105\") " Oct 30 08:00:23.214388 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.214359 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca" pod "711294e8-5614-4075-8ab9-5898ca529105" (UID: "711294e8-5614-4075-8ab9-5898ca529105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:23.214608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.214589 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711294e8-5614-4075-8ab9-5898ca529105-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:23.214946 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.214920 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca" pod "9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" (UID: "9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:23.217303 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.217274 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv" pod "711294e8-5614-4075-8ab9-5898ca529105" (UID: "711294e8-5614-4075-8ab9-5898ca529105"). InnerVolumeSpecName "kube-api-access-qk8kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:23.219913 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.219879 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq" pod "9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" (UID: "9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef"). InnerVolumeSpecName "kube-api-access-b8lmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:23.316398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.316038 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-maas-api-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:23.316398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.316080 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lmq\" (UniqueName: \"kubernetes.io/projected/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef-kube-api-access-b8lmq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:23.316398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.316098 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk8kv\" (UniqueName: \"kubernetes.io/projected/711294e8-5614-4075-8ab9-5898ca529105-kube-api-access-qk8kv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:23.785531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.785483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:23.918340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.918311 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" Oct 30 08:00:23.918340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.918316 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj" event={"ID":"9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef","Type":"ContainerDied","Data":"06ddaea116b1c17eb06afbe20549fa014fa89389a2faaa049b70e6e17634d7a7"} Oct 30 08:00:23.918914 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.918393 5894 scope.go:122] "RemoveContainer" containerID="c83408c2f63456c8b027e47ccf98b59c5d35e047e5ef26210174a9933c9217de" Oct 30 08:00:23.922833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.922811 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" Oct 30 08:00:23.922949 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.922852 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl" event={"ID":"711294e8-5614-4075-8ab9-5898ca529105","Type":"ContainerDied","Data":"71a77e07f1e4ca934f3257cc9532b89630112cd77ab3f2d6aad4491f5b3086d9"} Oct 30 08:00:23.932590 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.932569 5894 scope.go:122] "RemoveContainer" containerID="5c6646c5f6d373d005c84a478bda227cd0568a152c2bc9d4afc7f388641b3df4" Oct 30 08:00:23.989828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:23.989402 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:24.005241 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.002938 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-4b0e81-29889120-xd7cl"] Oct 30 08:00:24.034842 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.034806 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:24.042831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.042754 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-b096ec-29889120-pt7lj"] Oct 30 08:00:24.096847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.096811 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:24.133394 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.133357 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\") pod \"9e445752-1cf8-43cc-b495-61da204d5ee6\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " Oct 30 08:00:24.133551 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.133499 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\" (UniqueName: \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\") pod \"9e445752-1cf8-43cc-b495-61da204d5ee6\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " Oct 30 08:00:24.133551 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.133522 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\") pod \"9e445752-1cf8-43cc-b495-61da204d5ee6\" (UID: \"9e445752-1cf8-43cc-b495-61da204d5ee6\") " Oct 30 08:00:24.133871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.133834 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca" pod "9e445752-1cf8-43cc-b495-61da204d5ee6" (UID: "9e445752-1cf8-43cc-b495-61da204d5ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:24.136189 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.136163 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz" pod "9e445752-1cf8-43cc-b495-61da204d5ee6" (UID: "9e445752-1cf8-43cc-b495-61da204d5ee6"). InnerVolumeSpecName "kube-api-access-krmgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:24.141544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.138810 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca" pod "9e445752-1cf8-43cc-b495-61da204d5ee6" (UID: "9e445752-1cf8-43cc-b495-61da204d5ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:24.235071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.235035 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:24.235071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.235068 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmgz\" (UniqueName: \"kubernetes.io/projected/9e445752-1cf8-43cc-b495-61da204d5ee6-kube-api-access-krmgz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:24.235289 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.235082 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e445752-1cf8-43cc-b495-61da204d5ee6-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:24.434491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.434408 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" path="/var/lib/kubelet/pods/1c541641-0030-4fa0-8a8d-516668265dc3/volumes" Oct 30 08:00:24.434995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.434976 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" path="/var/lib/kubelet/pods/5c3c9121-b3c1-45e1-97d8-393910981f85/volumes" Oct 30 08:00:24.435441 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.435427 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="711294e8-5614-4075-8ab9-5898ca529105" path="/var/lib/kubelet/pods/711294e8-5614-4075-8ab9-5898ca529105/volumes" Oct 30 08:00:24.436189 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.436173 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" path="/var/lib/kubelet/pods/9a600d24-d431-4166-a41c-4f3c3012d5ad/volumes" Oct 30 08:00:24.436597 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.436582 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" path="/var/lib/kubelet/pods/9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5/volumes" Oct 30 08:00:24.436991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.436977 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" path="/var/lib/kubelet/pods/9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef/volumes" Oct 30 08:00:24.437661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.437642 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" path="/var/lib/kubelet/pods/abbe9d7a-6ea5-4ca5-8d46-863cec9cedea/volumes" Oct 30 08:00:24.438042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.438027 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" path="/var/lib/kubelet/pods/fa98e977-1e85-4da0-92e2-55e96732b578/volumes" Oct 30 08:00:24.935501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.935460 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" Oct 30 08:00:24.935974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.935470 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx" event={"ID":"9e445752-1cf8-43cc-b495-61da204d5ee6","Type":"ContainerDied","Data":"67b3cde7dafde9c453608dc4f78db81fcbfc6bd273cc022d2d52e8b559983f0b"} Oct 30 08:00:24.935974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:24.935610 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b3cde7dafde9c453608dc4f78db81fcbfc6bd273cc022d2d52e8b559983f0b" Oct 30 08:00:25.545590 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:25.545550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:27.195573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:27.195479 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:27.524329 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:27.524243 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:00:28.908330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:28.908287 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:30.576337 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:30.576289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:30.704191 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:30.704147 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:32.032377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:32.032334 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9"] Oct 30 08:00:32.044974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:32.044935 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a3366f68-x4wr9"] Oct 30 08:00:32.434202 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:32.434116 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73d760a-cdbb-4b6a-a894-ee593c427522" path="/var/lib/kubelet/pods/b73d760a-cdbb-4b6a-a894-ee593c427522/volumes" Oct 30 08:00:32.450722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:32.450689 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:32.995031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:32.994999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:33.343425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:33.343389 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:34.236168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:34.236122 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:34.830849 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:34.830811 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:00:35.811050 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811018 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:00:35.811841 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811823 5894 cpu_manager.go:414] "containerMap: removing container" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811842 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811851 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811856 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811865 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811871 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811880 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811885 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811892 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.811904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811897 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811912 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811917 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811925 5894 cpu_manager.go:414] "containerMap: removing container" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811929 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811935 5894 cpu_manager.go:414] "containerMap: removing container" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811939 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811945 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811950 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811958 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811962 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811968 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811972 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7e21f-3373-4018-ae96-582ce2ff9f59" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811986 5894 cpu_manager.go:414] "containerMap: removing container" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811990 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.811998 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812004 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2feccda1-6ff1-4e55-b1ae-28e499ac0f42" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812012 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812017 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812027 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812031 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812042 5894 cpu_manager.go:414] "containerMap: removing container" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812047 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812056 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812060 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812068 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812072 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812081 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812086 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812094 5894 cpu_manager.go:414] "containerMap: removing container" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812099 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812107 5894 cpu_manager.go:414] "containerMap: removing container" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812111 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812118 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812123 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812128 5894 cpu_manager.go:414] "containerMap: removing container" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812133 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812139 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9e445752-1cf8-43cc-b495-61da204d5ee6" containerName="revoke-keys" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812144 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e445752-1cf8-43cc-b495-61da204d5ee6" containerName="revoke-keys" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812159 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812164 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812178 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812182 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.812171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812190 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812194 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812298 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec31c06-c07b-4d87-852a-cad68487b55b" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812304 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812310 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812317 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaac19a2-19af-431e-9553-4eaea056ed79" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812323 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812329 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812335 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812342 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c415f0-2dcb-4da0-b3eb-f26abdb53126" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812349 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812355 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812361 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812366 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812370 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812375 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812381 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eced14c-1e20-4a1e-bc1b-bc705c0dc0de" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812389 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812394 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812401 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812407 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812412 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbf43d1-0ecf-4f36-9647-29eb95c3c9ef" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812416 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812421 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812425 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812430 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812434 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812439 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad9b3c2-06a2-4f2b-b995-69fbfbf1b1d5" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812445 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbe9d7a-6ea5-4ca5-8d46-863cec9cedea" containerName="cleanup" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812453 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e445752-1cf8-43cc-b495-61da204d5ee6" containerName="revoke-keys" Oct 30 08:00:35.813509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.812460 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:00:35.816290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.816266 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:35.835995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.835967 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:00:35.860695 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.844794 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2d8539a1-serving-cert\"" Oct 30 08:00:35.902586 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.902545 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:35.902779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.902598 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:35.902779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.902746 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:35.902906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.902893 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcckr\" (UniqueName: \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:35.913178 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:35.913146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.019694 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcckr\" (UniqueName: \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.019886 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.019930 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.019990 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:36.020277 5894 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-serving-cert: secret "maas-api-e2e-mt-2d8539a1-serving-cert" not found Oct 30 08:00:36.021146 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:36.020352 5894 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls podName:55af87cf-850b-4613-b22c-471d18938461 nodeName:}" failed. No retries permitted until 2026-10-30 08:00:36.520327865 +0000 UTC m=+3800.211245303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls") pod "maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" (UID: "55af87cf-850b-4613-b22c-471d18938461") : secret "maas-api-e2e-mt-2d8539a1-serving-cert" not found Oct 30 08:00:36.022745 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.022413 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.058407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.058373 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.062704 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.062549 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:36.066613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.066573 5894 generic.go:356] "Generic (PLEG): container finished" podID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerID="7c231430f8df26fc70948574e883499113e624bf0e6e9f9989748434effefdb1" exitCode=0 Oct 30 08:00:36.066903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.066875 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" event={"ID":"d7fe8038-db5d-4ce7-85df-584226b4e777","Type":"ContainerDied","Data":"7c231430f8df26fc70948574e883499113e624bf0e6e9f9989748434effefdb1"} Oct 30 08:00:36.068154 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.068029 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcckr\" (UniqueName: \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.160611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.160577 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 08:00:36.179796 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.179758 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:36.325527 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.325423 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\") pod \"d7fe8038-db5d-4ce7-85df-584226b4e777\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " Oct 30 08:00:36.325732 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.325718 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\" (UniqueName: \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\") pod \"d7fe8038-db5d-4ce7-85df-584226b4e777\" (UID: \"d7fe8038-db5d-4ce7-85df-584226b4e777\") " Oct 30 08:00:36.325897 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.325875 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume" pod "d7fe8038-db5d-4ce7-85df-584226b4e777" (UID: "d7fe8038-db5d-4ce7-85df-584226b4e777"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:36.326229 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.326213 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fe8038-db5d-4ce7-85df-584226b4e777-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:36.327975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.327944 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b" pod "d7fe8038-db5d-4ce7-85df-584226b4e777" (UID: "d7fe8038-db5d-4ce7-85df-584226b4e777"). InnerVolumeSpecName "kube-api-access-zpm2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:36.427748 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.427711 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpm2b\" (UniqueName: \"kubernetes.io/projected/d7fe8038-db5d-4ce7-85df-584226b4e777-kube-api-access-zpm2b\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:36.528937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.528901 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.531435 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.531407 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") pod \"maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:36.791922 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:36.791870 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:37.116733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.116580 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" event={"ID":"d7fe8038-db5d-4ce7-85df-584226b4e777","Type":"ContainerDied","Data":"2932e435547b94929092609b1fcfa426063bc7e0d114abacad4f9b774e097248"} Oct 30 08:00:37.117232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.116749 5894 scope.go:122] "RemoveContainer" containerID="7c231430f8df26fc70948574e883499113e624bf0e6e9f9989748434effefdb1" Oct 30 08:00:37.117232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.116994 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6" Oct 30 08:00:37.124278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.124248 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:00:37.131303 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:00:37.131262 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55af87cf_850b_4613_b22c_471d18938461.slice/crio-49ad464be3c993bfc941dac3d3756cee28657936dc52f28b087e816565c8db07 WatchSource:0}: Error finding container 49ad464be3c993bfc941dac3d3756cee28657936dc52f28b087e816565c8db07: Status 404 returned error can't find the container with id 49ad464be3c993bfc941dac3d3756cee28657936dc52f28b087e816565c8db07 Oct 30 08:00:37.136944 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.136914 5894 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 08:00:37.173336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.173304 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 08:00:37.176431 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.176407 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-268576-58db974c47-f4kc6"] Oct 30 08:00:37.498616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.498519 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:37.734927 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:37.734885 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:38.126740 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:38.126688 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" event={"ID":"55af87cf-850b-4613-b22c-471d18938461","Type":"ContainerStarted","Data":"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b"} Oct 30 08:00:38.127238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:38.126758 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" event={"ID":"55af87cf-850b-4613-b22c-471d18938461","Type":"ContainerStarted","Data":"49ad464be3c993bfc941dac3d3756cee28657936dc52f28b087e816565c8db07"} Oct 30 08:00:38.127238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:38.126787 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:38.170706 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:38.170577 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" podStartSLOduration=2.7339761989999998 podStartE2EDuration="3.170560264s" totalImagesPullingTime="436.584065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:00:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:00:38.169039997 +0000 UTC m=+3801.859957425" watchObservedRunningTime="2026-10-30 08:00:38.170560264 +0000 UTC m=+3801.861477701" Oct 30 08:00:38.433010 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:38.432931 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" path="/var/lib/kubelet/pods/d7fe8038-db5d-4ce7-85df-584226b4e777/volumes" Oct 30 08:00:39.452603 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:39.452539 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:41.164997 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:41.164958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:43.040126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:43.040085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:44.141148 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:44.141116 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:00:44.770564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:44.770520 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:46.159135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.159097 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 08:00:46.159951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.159917 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerName="maas-api" containerID="cri-o://3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a" gracePeriod=30 Oct 30 08:00:46.211870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.211813 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.89:8443/health\": dial tcp 10.134.1.89:8443: connect: connection refused" Oct 30 08:00:46.260651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.260595 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:46.485821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.485791 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 08:00:46.597661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.595497 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\") pod \"af913dfa-4d24-407d-8e22-cb03ead91b61\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " Oct 30 08:00:46.597661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.595732 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\" (UniqueName: \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\") pod \"af913dfa-4d24-407d-8e22-cb03ead91b61\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " Oct 30 08:00:46.597661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.595807 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\") pod \"af913dfa-4d24-407d-8e22-cb03ead91b61\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " Oct 30 08:00:46.597661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.595866 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\") pod \"af913dfa-4d24-407d-8e22-cb03ead91b61\" (UID: \"af913dfa-4d24-407d-8e22-cb03ead91b61\") " Oct 30 08:00:46.609657 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.601771 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca" pod "af913dfa-4d24-407d-8e22-cb03ead91b61" (UID: "af913dfa-4d24-407d-8e22-cb03ead91b61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:46.672222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.672162 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs" pod "af913dfa-4d24-407d-8e22-cb03ead91b61" (UID: "af913dfa-4d24-407d-8e22-cb03ead91b61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:46.672473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.672441 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg" pod "af913dfa-4d24-407d-8e22-cb03ead91b61" (UID: "af913dfa-4d24-407d-8e22-cb03ead91b61"). InnerVolumeSpecName "kube-api-access-8g6pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:46.684303 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.684247 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls" pod "af913dfa-4d24-407d-8e22-cb03ead91b61" (UID: "af913dfa-4d24-407d-8e22-cb03ead91b61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:46.783719 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.783686 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-maas-api-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:46.783931 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.783915 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6pg\" (UniqueName: \"kubernetes.io/projected/af913dfa-4d24-407d-8e22-cb03ead91b61-kube-api-access-8g6pg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:46.784040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.784028 5894 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af913dfa-4d24-407d-8e22-cb03ead91b61-openshift-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:46.784152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:46.784141 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af913dfa-4d24-407d-8e22-cb03ead91b61-metrics-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:47.017659 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.010653 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:47.244723 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.242694 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:47.262674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.261606 5894 generic.go:356] "Generic (PLEG): container finished" podID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerID="3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a" exitCode=0 Oct 30 08:00:47.262674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.261865 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" event={"ID":"af913dfa-4d24-407d-8e22-cb03ead91b61","Type":"ContainerDied","Data":"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a"} Oct 30 08:00:47.262674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.261927 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" event={"ID":"af913dfa-4d24-407d-8e22-cb03ead91b61","Type":"ContainerDied","Data":"d82ae97f3b8c71f7c648c15af894c9a7fe250187b8c5aeb2c500de21a4c94e93"} Oct 30 08:00:47.262674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.261961 5894 scope.go:122] "RemoveContainer" containerID="3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a" Oct 30 08:00:47.262674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.262134 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm" Oct 30 08:00:47.338739 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.338695 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn"] Oct 30 08:00:47.421929 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.421228 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-352fb3-mpskn"] Oct 30 08:00:47.481740 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.481590 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:47.509714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.502893 5894 scope.go:122] "RemoveContainer" containerID="3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a" Oct 30 08:00:47.509714 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:47.503493 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a\": container with ID starting with 3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a not found: ID does not exist" containerID="3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a" Oct 30 08:00:47.509714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.503553 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a"} err="failed to get container status \"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a\": rpc error: code = NotFound desc = could not find container \"3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a\": container with ID starting with 3bf9ea98675cd312c2e607fa5b1fd7e6ce5d2e0ac82b1d357cc5dc357fa1725a not found: ID does not exist" Oct 30 08:00:47.532661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.526689 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 08:00:47.566690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.550608 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-352fb3-55cd8b5b88-rqtfm"] Oct 30 08:00:47.818656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:47.818270 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:48.454788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:48.454744 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9ef108-46fa-458b-af70-90065ca8c203" path="/var/lib/kubelet/pods/8c9ef108-46fa-458b-af70-90065ca8c203/volumes" Oct 30 08:00:48.455333 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:48.455267 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" path="/var/lib/kubelet/pods/af913dfa-4d24-407d-8e22-cb03ead91b61/volumes" Oct 30 08:00:49.326678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.326627 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 08:00:49.326933 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.326863 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" podUID="ee13a129-5502-4364-9d45-2681daca4624" containerName="maas-api" containerID="cri-o://7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45" gracePeriod=30 Oct 30 08:00:49.437146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.437043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:49.705749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.705349 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 08:00:49.874709 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.872471 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\") pod \"ee13a129-5502-4364-9d45-2681daca4624\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " Oct 30 08:00:49.874709 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.872550 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\") pod \"ee13a129-5502-4364-9d45-2681daca4624\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " Oct 30 08:00:49.874709 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.872601 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\") pod \"ee13a129-5502-4364-9d45-2681daca4624\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " Oct 30 08:00:49.874709 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.872896 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\" (UniqueName: \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\") pod \"ee13a129-5502-4364-9d45-2681daca4624\" (UID: \"ee13a129-5502-4364-9d45-2681daca4624\") " Oct 30 08:00:49.894462 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.891913 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8" pod "ee13a129-5502-4364-9d45-2681daca4624" (UID: "ee13a129-5502-4364-9d45-2681daca4624"). InnerVolumeSpecName "kube-api-access-px6f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:00:49.897233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.896698 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs" pod "ee13a129-5502-4364-9d45-2681daca4624" (UID: "ee13a129-5502-4364-9d45-2681daca4624"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:49.900774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.899797 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca" pod "ee13a129-5502-4364-9d45-2681daca4624" (UID: "ee13a129-5502-4364-9d45-2681daca4624"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:00:49.902553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.902483 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls" pod "ee13a129-5502-4364-9d45-2681daca4624" (UID: "ee13a129-5502-4364-9d45-2681daca4624"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:00:49.974246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.974209 5894 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee13a129-5502-4364-9d45-2681daca4624-openshift-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:49.974246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.974244 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6f8\" (UniqueName: \"kubernetes.io/projected/ee13a129-5502-4364-9d45-2681daca4624-kube-api-access-px6f8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:49.974246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.974255 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-metrics-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:49.974516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:49.974265 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee13a129-5502-4364-9d45-2681daca4624-maas-api-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:00:50.110497 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.110462 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:50.244440 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.244350 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:50.279832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.279784 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg"] Oct 30 08:00:50.287097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.287058 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-268576-5lzmg"] Oct 30 08:00:50.297954 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.297918 5894 generic.go:356] "Generic (PLEG): container finished" podID="ee13a129-5502-4364-9d45-2681daca4624" containerID="7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45" exitCode=0 Oct 30 08:00:50.298146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.297953 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" event={"ID":"ee13a129-5502-4364-9d45-2681daca4624","Type":"ContainerDied","Data":"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45"} Oct 30 08:00:50.298146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.297997 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" Oct 30 08:00:50.298146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.298053 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6" event={"ID":"ee13a129-5502-4364-9d45-2681daca4624","Type":"ContainerDied","Data":"ed202fb94ef6b8efab678fe0988d90fc7b80a572cdf16c7e95cda752f2763d9b"} Oct 30 08:00:50.298146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.298082 5894 scope.go:122] "RemoveContainer" containerID="7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45" Oct 30 08:00:50.310533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.310495 5894 scope.go:122] "RemoveContainer" containerID="7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45" Oct 30 08:00:50.311013 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:00:50.310991 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45\": container with ID starting with 7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45 not found: ID does not exist" containerID="7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45" Oct 30 08:00:50.311112 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.311023 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45"} err="failed to get container status \"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45\": rpc error: code = NotFound desc = could not find container \"7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45\": container with ID starting with 7b3a517b91a6fe9b77fc6411e6a6785a4bc3aef5c2a5fb53bb5efa752a808e45 not found: ID does not exist" Oct 30 08:00:50.359791 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.359755 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 08:00:50.364732 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.364700 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-268576-65d5dcdd89-nqkr6"] Oct 30 08:00:50.433268 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.433235 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc6cbd5-f519-4d14-82f1-f2e0eaf123cf" path="/var/lib/kubelet/pods/adc6cbd5-f519-4d14-82f1-f2e0eaf123cf/volumes" Oct 30 08:00:50.433783 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.433767 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee13a129-5502-4364-9d45-2681daca4624" path="/var/lib/kubelet/pods/ee13a129-5502-4364-9d45-2681daca4624/volumes" Oct 30 08:00:50.789804 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:50.789750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:52.035623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:52.035582 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:53.759074 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:53.759025 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:54.372816 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:54.372779 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:54.488224 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:54.488170 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:54.736134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:54.736031 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:55.424218 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:55.424184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:57.205015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:57.204971 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:57.324487 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:57.324449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:57.524351 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:57.524254 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 08:00:57.524535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:57.524513 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" containerID="cri-o://838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9" gracePeriod=30 Oct 30 08:00:57.592876 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:57.592837 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:58.546891 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:58.546855 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:58.667423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:58.667383 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:58.899378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:58.899289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:00:59.105656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:00:59.102379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:00.841775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:00.841737 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:02.446318 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.446275 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:02.804391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.804362 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 08:01:02.873800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.873766 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.873981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.873808 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.873981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.873830 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.873981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.873869 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.873981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.873937 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.874221 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874017 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.874221 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874043 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.874335 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874248 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.874335 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874301 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\") pod \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\" (UID: \"cd898a2f-7a0f-47b3-b134-cdc3bd15590a\") " Oct 30 08:01:02.874458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874354 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:02.874458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874270 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:02.874591 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874504 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:02.874701 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874596 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:02.874701 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874682 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:02.874984 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874962 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.875067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.874989 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.875067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.875003 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.875067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.875019 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.875067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.875032 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.876463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.876437 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:02.876693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.876662 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:01:02.876791 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.876723 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:01:02.876971 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.876950 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt" pod "cd898a2f-7a0f-47b3-b134-cdc3bd15590a" (UID: "cd898a2f-7a0f-47b3-b134-cdc3bd15590a"). InnerVolumeSpecName "kube-api-access-t9kbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:02.976114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.976014 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.976114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.976049 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9kbt\" (UniqueName: \"kubernetes.io/projected/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-kube-api-access-t9kbt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.976114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.976065 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:02.976114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:02.976077 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd898a2f-7a0f-47b3-b134-cdc3bd15590a-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:03.408224 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.408190 5894 generic.go:356] "Generic (PLEG): container finished" podID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerID="838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9" exitCode=0 Oct 30 08:01:03.408412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.408238 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" Oct 30 08:01:03.408412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.408261 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerDied","Data":"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9"} Oct 30 08:01:03.408412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.408306 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k" event={"ID":"cd898a2f-7a0f-47b3-b134-cdc3bd15590a","Type":"ContainerDied","Data":"653f8b0baefa4c788c54b2e6fc492971d921636713d357db07b930a0bf9bfa68"} Oct 30 08:01:03.408412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.408328 5894 scope.go:122] "RemoveContainer" containerID="838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9" Oct 30 08:01:03.420899 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.420867 5894 scope.go:122] "RemoveContainer" containerID="c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5" Oct 30 08:01:03.431913 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.431889 5894 scope.go:122] "RemoveContainer" containerID="838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9" Oct 30 08:01:03.432231 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:03.432213 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9\": container with ID starting with 838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9 not found: ID does not exist" containerID="838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9" Oct 30 08:01:03.432284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.432238 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9"} err="failed to get container status \"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9\": rpc error: code = NotFound desc = could not find container \"838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9\": container with ID starting with 838177e0a3a04c9cbdbc0f7ae2709b325695bba53176b111beb084e39d5f21c9 not found: ID does not exist" Oct 30 08:01:03.432284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.432261 5894 scope.go:122] "RemoveContainer" containerID="c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5" Oct 30 08:01:03.432485 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:03.432469 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5\": container with ID starting with c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5 not found: ID does not exist" containerID="c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5" Oct 30 08:01:03.432533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.432491 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5"} err="failed to get container status \"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5\": rpc error: code = NotFound desc = could not find container \"c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5\": container with ID starting with c7079793396976b9869f3ec1afe54fa6466f1204c90203864b962402ca7529f5 not found: ID does not exist" Oct 30 08:01:03.462830 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.462798 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 08:01:03.475834 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:03.475794 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-268576-openshift-default-d598b979c-d2z8k"] Oct 30 08:01:04.252202 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:04.252164 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:04.433231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:04.433197 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" path="/var/lib/kubelet/pods/cd898a2f-7a0f-47b3-b134-cdc3bd15590a/volumes" Oct 30 08:01:04.838002 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:04.837959 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:01:05.345803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.345764 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:01:05.346654 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346614 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346656 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346678 5894 cpu_manager.go:414] "containerMap: removing container" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346683 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346694 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346699 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346711 5894 cpu_manager.go:414] "containerMap: removing container" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.346714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346717 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346726 5894 cpu_manager.go:414] "containerMap: removing container" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerName="maas-api" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346730 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerName="maas-api" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346752 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ee13a129-5502-4364-9d45-2681daca4624" containerName="maas-api" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346756 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee13a129-5502-4364-9d45-2681daca4624" containerName="maas-api" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346770 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346774 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346786 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346790 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346798 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346802 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a600d24-d431-4166-a41c-4f3c3012d5ad" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346810 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346814 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c541641-0030-4fa0-8a8d-516668265dc3" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346830 5894 cpu_manager.go:414] "containerMap: removing container" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346834 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="711294e8-5614-4075-8ab9-5898ca529105" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346925 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c3c9121-b3c1-45e1-97d8-393910981f85" containerName="cleanup" Oct 30 08:01:05.346935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346935 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fe8038-db5d-4ce7-85df-584226b4e777" containerName="payload-pre-processing" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346943 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346951 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e99876-ea97-489e-9728-25fa00c726d6" containerName="cleanup" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346958 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98e977-1e85-4da0-92e2-55e96732b578" containerName="cleanup" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346964 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd898a2f-7a0f-47b3-b134-cdc3bd15590a" containerName="istio-proxy" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346971 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="af913dfa-4d24-407d-8e22-cb03ead91b61" containerName="maas-api" Oct 30 08:01:05.347451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.346979 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee13a129-5502-4364-9d45-2681daca4624" containerName="maas-api" Oct 30 08:01:05.352102 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.352076 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.376383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.376346 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:01:05.385870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.385834 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-bd38cf-openshift-default-dockercfg-rphjj\"" Oct 30 08:01:05.415827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.415796 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.415986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.415844 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.415986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.415915 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrt9\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.415992 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.416023 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.416058 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.416115 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.416275 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.416383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.416368 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517719 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517775 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517802 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrt9\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517827 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517882 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.517952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517930 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.517953 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518119 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518190 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518194 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518312 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518470 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518494 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.518845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.518822 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.520163 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.520144 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.520764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.520741 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.543463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.543429 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.543684 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.543658 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrt9\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\") pod \"e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.585610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.585570 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:05.666574 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.666485 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:05.731558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.731440 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:05.842169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.842121 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7"] Oct 30 08:01:05.851215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.851187 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:05.868374 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.868264 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:01:05.868762 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.868723 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:01:05.869225 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.869197 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7"] Oct 30 08:01:05.926521 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.926445 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h747r\" (UniqueName: \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:05.926521 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.926508 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:05.926763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.926610 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:05.984812 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:01:05.984780 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4e6f71_e06b_4fb4_9228_f35735f6bf2e.slice/crio-11d968aa2af55603c331c45db09b7f22519571564705e087534ba24dfc4f0265 WatchSource:0}: Error finding container 11d968aa2af55603c331c45db09b7f22519571564705e087534ba24dfc4f0265: Status 404 returned error can't find the container with id 11d968aa2af55603c331c45db09b7f22519571564705e087534ba24dfc4f0265 Oct 30 08:01:05.985232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.985208 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:01:05.988024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.987993 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:01:05.988124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.988058 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:01:05.988124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:05.988086 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:01:06.028111 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.028077 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h747r\" (UniqueName: \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.028285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.028136 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.028285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.028177 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.028946 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.028922 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.030605 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.030584 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.053279 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.053247 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h747r\" (UniqueName: \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\") pod \"maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.166812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.166773 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:06.398595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.398558 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:06.442703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.442560 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" event={"ID":"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e","Type":"ContainerStarted","Data":"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303"} Oct 30 08:01:06.442703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.442644 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" event={"ID":"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e","Type":"ContainerStarted","Data":"11d968aa2af55603c331c45db09b7f22519571564705e087534ba24dfc4f0265"} Oct 30 08:01:06.483126 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:01:06.483090 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243cc756_febc_4d7c_bd1a_8624fcc2e5e4.slice/crio-6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90 WatchSource:0}: Error finding container 6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90: Status 404 returned error can't find the container with id 6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90 Oct 30 08:01:06.484093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.484065 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7"] Oct 30 08:01:06.496732 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.496674 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podStartSLOduration=1.49666155 podStartE2EDuration="1.49666155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:06.495369433 +0000 UTC m=+3830.186286870" watchObservedRunningTime="2026-10-30 08:01:06.49666155 +0000 UTC m=+3830.187578987" Oct 30 08:01:06.667525 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:06.667494 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:07.451986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:07.451947 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" event={"ID":"243cc756-febc-4d7c-bd1a-8624fcc2e5e4","Type":"ContainerStarted","Data":"9f9693bd1ab56bed9fec0b50687b7175d07edc715b701255c93fc5fab7139f06"} Oct 30 08:01:07.452446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:07.452017 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" event={"ID":"243cc756-febc-4d7c-bd1a-8624fcc2e5e4","Type":"ContainerStarted","Data":"6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90"} Oct 30 08:01:07.510364 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:07.510302 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" podStartSLOduration=2.510287696 podStartE2EDuration="2.510287696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:07.506814681 +0000 UTC m=+3831.197732119" watchObservedRunningTime="2026-10-30 08:01:07.510287696 +0000 UTC m=+3831.201205140" Oct 30 08:01:07.669172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:07.669136 5894 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" start-of-body= Oct 30 08:01:07.669372 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:07.669206 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" Oct 30 08:01:08.204015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:08.203975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:08.461796 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:08.461699 5894 generic.go:356] "Generic (PLEG): container finished" podID="243cc756-febc-4d7c-bd1a-8624fcc2e5e4" containerID="9f9693bd1ab56bed9fec0b50687b7175d07edc715b701255c93fc5fab7139f06" exitCode=0 Oct 30 08:01:08.462286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:08.461790 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" event={"ID":"243cc756-febc-4d7c-bd1a-8624fcc2e5e4","Type":"ContainerDied","Data":"9f9693bd1ab56bed9fec0b50687b7175d07edc715b701255c93fc5fab7139f06"} Oct 30 08:01:08.667221 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:08.667183 5894 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" start-of-body= Oct 30 08:01:08.667413 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:08.667255 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" Oct 30 08:01:09.667052 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.667022 5894 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" start-of-body= Oct 30 08:01:09.667440 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.667081 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" Oct 30 08:01:09.697192 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.697158 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:09.814276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.814222 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\") pod \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " Oct 30 08:01:09.814479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.814309 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\") pod \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " Oct 30 08:01:09.814479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.814416 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\" (UniqueName: \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\") pod \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\" (UID: \"243cc756-febc-4d7c-bd1a-8624fcc2e5e4\") " Oct 30 08:01:09.814811 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.814775 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca" pod "243cc756-febc-4d7c-bd1a-8624fcc2e5e4" (UID: "243cc756-febc-4d7c-bd1a-8624fcc2e5e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:09.815081 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.815053 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:09.816613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.816593 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca" pod "243cc756-febc-4d7c-bd1a-8624fcc2e5e4" (UID: "243cc756-febc-4d7c-bd1a-8624fcc2e5e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:01:09.816726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.816668 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r" pod "243cc756-febc-4d7c-bd1a-8624fcc2e5e4" (UID: "243cc756-febc-4d7c-bd1a-8624fcc2e5e4"). InnerVolumeSpecName "kube-api-access-h747r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:09.916774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.916741 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:09.916774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:09.916771 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h747r\" (UniqueName: \"kubernetes.io/projected/243cc756-febc-4d7c-bd1a-8624fcc2e5e4-kube-api-access-h747r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:10.033461 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.033426 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:10.503761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.503730 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" Oct 30 08:01:10.503974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.503757 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7" event={"ID":"243cc756-febc-4d7c-bd1a-8624fcc2e5e4","Type":"ContainerDied","Data":"6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90"} Oct 30 08:01:10.503974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.503812 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b535a7db366699cb062a6e9cb0b9bb7d384dffeaac372337ad86994c49a1d90" Oct 30 08:01:10.667363 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.667322 5894 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" start-of-body= Oct 30 08:01:10.667803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.667383 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" Oct 30 08:01:10.803660 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:10.798574 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:01:11.670542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:11.670508 5894 patch_prober.go:29] interesting pod/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" start-of-body= Oct 30 08:01:11.671186 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:11.671157 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.135:15021/healthz/ready\": dial tcp 10.134.1.135:15021: connect: connection refused" Oct 30 08:01:12.101664 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:12.101614 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:12.730874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:12.730828 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:12.731466 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:12.731439 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:12.732527 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:12.732504 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:01:14.038267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:14.038218 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:15.738989 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:15.738952 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:16.393549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:16.393511 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 08:01:16.434020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:16.433986 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 08:01:16.571042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:16.570993 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:16.740274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:16.740186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:17.032974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:17.032931 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8"] Oct 30 08:01:17.037036 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:17.036997 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4af27a0-pq8z8"] Oct 30 08:01:17.520585 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:17.520494 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:18.432702 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:18.432667 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bc67126-aa1c-4504-a7f1-b4f361af286f" path="/var/lib/kubelet/pods/9bc67126-aa1c-4504-a7f1-b4f361af286f/volumes" Oct 30 08:01:19.351377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:19.351335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:21.168379 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:21.168346 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:21.343586 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:21.343549 5894 scope.go:122] "RemoveContainer" containerID="b819871a6c1e9d857ba2487eded82e20a187ed3490ce2c9f83582f33fc36880b" Oct 30 08:01:21.354907 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:21.354881 5894 scope.go:122] "RemoveContainer" containerID="861e988edb17746371a8498b6ccc75373737b35bd5576a82e77166a6a55add6f" Oct 30 08:01:21.394199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:21.394163 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" containerID="cri-o://b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2" gracePeriod=25 Oct 30 08:01:21.435056 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:21.435022 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" containerID="cri-o://8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b" gracePeriod=25 Oct 30 08:01:22.849304 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:22.849263 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:23.902389 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:23.902350 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:23.902832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:23.902363 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:24.333435 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:24.333393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:24.911913 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:24.911873 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:25.021496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:25.021448 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7"] Oct 30 08:01:25.035466 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:25.035418 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-a91a83-5fpq7"] Oct 30 08:01:25.055968 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:25.055928 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:25.526186 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:25.526146 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:26.433117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:26.433073 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243cc756-febc-4d7c-bd1a-8624fcc2e5e4" path="/var/lib/kubelet/pods/243cc756-febc-4d7c-bd1a-8624fcc2e5e4/volumes" Oct 30 08:01:26.838737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:26.838668 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:28.928195 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:28.928159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:30.722354 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:30.722303 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:31.356331 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:31.356290 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:32.188143 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:32.188096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:32.357731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:32.357691 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:32.750216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:32.750181 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:32.928182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:32.928134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:33.132721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:33.132679 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:33.479969 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:33.479879 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:33.902442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:33.902395 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:33.902663 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:33.902405 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:34.441200 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:34.441159 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:34.578241 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:34.578200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:34.838590 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:34.838546 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:01:34.861477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:34.861416 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:35.395377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.395342 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb"] Oct 30 08:01:35.396107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.396079 5894 cpu_manager.go:414] "containerMap: removing container" podUID="243cc756-febc-4d7c-bd1a-8624fcc2e5e4" containerName="revoke-keys" Oct 30 08:01:35.396107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.396101 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243cc756-febc-4d7c-bd1a-8624fcc2e5e4" containerName="revoke-keys" Oct 30 08:01:35.396292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.396249 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="243cc756-febc-4d7c-bd1a-8624fcc2e5e4" containerName="revoke-keys" Oct 30 08:01:35.403157 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.403133 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.405285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.405240 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.405471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.405307 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.405552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.405474 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtv6w\" (UniqueName: \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.420757 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.420719 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb"] Oct 30 08:01:35.429991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.429959 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:01:35.430158 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.430023 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:01:35.506595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.506549 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.506877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.506626 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.506970 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.506864 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtv6w\" (UniqueName: \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.507415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.507394 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.509387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.509363 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.535256 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.535218 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtv6w\" (UniqueName: \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\") pod \"maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:35.718889 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:35.718806 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:36.028470 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:36.028435 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb"] Oct 30 08:01:36.028795 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:01:36.028771 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09754aae_6a10_4c29_ac00_a4ca7093633f.slice/crio-aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80 WatchSource:0}: Error finding container aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80: Status 404 returned error can't find the container with id aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80 Oct 30 08:01:36.599696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:36.591953 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:36.773480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:36.773391 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" event={"ID":"09754aae-6a10-4c29-ac00-a4ca7093633f","Type":"ContainerStarted","Data":"56dad326f4e01c3af7bd8703985f5d2e62d8eff121bc12ac8d475f5ed38ddb12"} Oct 30 08:01:36.773480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:36.773453 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" event={"ID":"09754aae-6a10-4c29-ac00-a4ca7093633f","Type":"ContainerStarted","Data":"aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80"} Oct 30 08:01:36.828141 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:36.828083 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" podStartSLOduration=1.828068126 podStartE2EDuration="1.828068126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:01:36.824939537 +0000 UTC m=+3860.515856979" watchObservedRunningTime="2026-10-30 08:01:36.828068126 +0000 UTC m=+3860.518985563" Oct 30 08:01:37.783032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:37.782999 5894 generic.go:356] "Generic (PLEG): container finished" podID="09754aae-6a10-4c29-ac00-a4ca7093633f" containerID="56dad326f4e01c3af7bd8703985f5d2e62d8eff121bc12ac8d475f5ed38ddb12" exitCode=0 Oct 30 08:01:37.783453 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:37.783097 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" event={"ID":"09754aae-6a10-4c29-ac00-a4ca7093633f","Type":"ContainerDied","Data":"56dad326f4e01c3af7bd8703985f5d2e62d8eff121bc12ac8d475f5ed38ddb12"} Oct 30 08:01:38.465477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:38.465436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:38.971157 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:38.971130 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:39.051417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.051327 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\") pod \"09754aae-6a10-4c29-ac00-a4ca7093633f\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " Oct 30 08:01:39.051417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.051386 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\") pod \"09754aae-6a10-4c29-ac00-a4ca7093633f\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " Oct 30 08:01:39.051704 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.051484 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\" (UniqueName: \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\") pod \"09754aae-6a10-4c29-ac00-a4ca7093633f\" (UID: \"09754aae-6a10-4c29-ac00-a4ca7093633f\") " Oct 30 08:01:39.051880 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.051841 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca" pod "09754aae-6a10-4c29-ac00-a4ca7093633f" (UID: "09754aae-6a10-4c29-ac00-a4ca7093633f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:39.052031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.051994 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:39.053801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.053771 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca" pod "09754aae-6a10-4c29-ac00-a4ca7093633f" (UID: "09754aae-6a10-4c29-ac00-a4ca7093633f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:01:39.053917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.053818 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w" pod "09754aae-6a10-4c29-ac00-a4ca7093633f" (UID: "09754aae-6a10-4c29-ac00-a4ca7093633f"). InnerVolumeSpecName "kube-api-access-qtv6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:39.152855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.152815 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09754aae-6a10-4c29-ac00-a4ca7093633f-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:39.152855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.152851 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtv6w\" (UniqueName: \"kubernetes.io/projected/09754aae-6a10-4c29-ac00-a4ca7093633f-kube-api-access-qtv6w\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:39.799210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.799178 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" Oct 30 08:01:39.799416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.799206 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb" event={"ID":"09754aae-6a10-4c29-ac00-a4ca7093633f","Type":"ContainerDied","Data":"aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80"} Oct 30 08:01:39.799416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:39.799270 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aade92295f389ec0f6fda6a178ae4ffa7055e6504dea019f5378b4d5b6a1fc80" Oct 30 08:01:40.409070 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:40.409033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:40.623723 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:40.623693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:41.724718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.724686 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 08:01:41.759942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.759911 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 08:01:41.787177 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787139 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\" (UniqueName: \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\") pod \"d3d9383d-2370-4553-bfb4-298acddff9d3\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " Oct 30 08:01:41.787380 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787210 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\") pod \"d3d9383d-2370-4553-bfb4-298acddff9d3\" (UID: \"d3d9383d-2370-4553-bfb4-298acddff9d3\") " Oct 30 08:01:41.787380 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787320 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\") pod \"4c0984ff-509b-4910-8f46-437583c2f376\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " Oct 30 08:01:41.787534 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787515 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\" (UniqueName: \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\") pod \"4c0984ff-509b-4910-8f46-437583c2f376\" (UID: \"4c0984ff-509b-4910-8f46-437583c2f376\") " Oct 30 08:01:41.787671 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787627 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume" pod "d3d9383d-2370-4553-bfb4-298acddff9d3" (UID: "d3d9383d-2370-4553-bfb4-298acddff9d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:41.787831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.787808 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume" pod "4c0984ff-509b-4910-8f46-437583c2f376" (UID: "4c0984ff-509b-4910-8f46-437583c2f376"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:01:41.788126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.788100 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3d9383d-2370-4553-bfb4-298acddff9d3-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:41.788223 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.788127 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c0984ff-509b-4910-8f46-437583c2f376-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:41.789762 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.789737 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp" pod "4c0984ff-509b-4910-8f46-437583c2f376" (UID: "4c0984ff-509b-4910-8f46-437583c2f376"). InnerVolumeSpecName "kube-api-access-s4fhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:41.789992 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.789973 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7" pod "d3d9383d-2370-4553-bfb4-298acddff9d3" (UID: "d3d9383d-2370-4553-bfb4-298acddff9d3"). InnerVolumeSpecName "kube-api-access-v79v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:01:41.818942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.818907 5894 generic.go:356] "Generic (PLEG): container finished" podID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerID="b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2" exitCode=0 Oct 30 08:01:41.819274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.818975 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" Oct 30 08:01:41.819274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.818997 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" event={"ID":"d3d9383d-2370-4553-bfb4-298acddff9d3","Type":"ContainerDied","Data":"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2"} Oct 30 08:01:41.819274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.819074 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n" event={"ID":"d3d9383d-2370-4553-bfb4-298acddff9d3","Type":"ContainerDied","Data":"d13b9940d1ed38d701d8b8d67bf4f4eee9dbc5db0aa886bf9eb13c049e564bed"} Oct 30 08:01:41.819274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.819107 5894 scope.go:122] "RemoveContainer" containerID="b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2" Oct 30 08:01:41.823069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.823037 5894 generic.go:356] "Generic (PLEG): container finished" podID="4c0984ff-509b-4910-8f46-437583c2f376" containerID="8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b" exitCode=0 Oct 30 08:01:41.823199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.823127 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" event={"ID":"4c0984ff-509b-4910-8f46-437583c2f376","Type":"ContainerDied","Data":"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b"} Oct 30 08:01:41.823199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.823170 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" event={"ID":"4c0984ff-509b-4910-8f46-437583c2f376","Type":"ContainerDied","Data":"b691fa3eeac657aff5d563279c993f18459a865dcc6ef42902c5d2006eba1527"} Oct 30 08:01:41.823328 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.823244 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq" Oct 30 08:01:41.847406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.847376 5894 scope.go:122] "RemoveContainer" containerID="b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2" Oct 30 08:01:41.848788 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:41.848019 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2\": container with ID starting with b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2 not found: ID does not exist" containerID="b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2" Oct 30 08:01:41.848788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.848064 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2"} err="failed to get container status \"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2\": rpc error: code = NotFound desc = could not find container \"b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2\": container with ID starting with b507d7ca0588fa4109276551f95be5e4ab676c94de76de897071e5e4e15ea6a2 not found: ID does not exist" Oct 30 08:01:41.848788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.848090 5894 scope.go:122] "RemoveContainer" containerID="8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b" Oct 30 08:01:41.860788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.860763 5894 scope.go:122] "RemoveContainer" containerID="8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b" Oct 30 08:01:41.861135 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:01:41.861107 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b\": container with ID starting with 8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b not found: ID does not exist" containerID="8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b" Oct 30 08:01:41.861204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.861149 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b"} err="failed to get container status \"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b\": rpc error: code = NotFound desc = could not find container \"8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b\": container with ID starting with 8111656999a4345e7524c239759ad3cc8f522f2974133563b74953cb4c7d295b not found: ID does not exist" Oct 30 08:01:41.889542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.889504 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4fhp\" (UniqueName: \"kubernetes.io/projected/4c0984ff-509b-4910-8f46-437583c2f376-kube-api-access-s4fhp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:41.889542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.889540 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79v7\" (UniqueName: \"kubernetes.io/projected/d3d9383d-2370-4553-bfb4-298acddff9d3-kube-api-access-v79v7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:01:41.897387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.897357 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 08:01:41.900754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.900724 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-a91a83-5f658cfc58-c8b6n"] Oct 30 08:01:41.939196 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.939098 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 08:01:41.942862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:41.942829 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-a91a83-76997f6f55-g8gwq"] Oct 30 08:01:42.163295 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:42.163263 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:42.432607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:42.432564 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c0984ff-509b-4910-8f46-437583c2f376" path="/var/lib/kubelet/pods/4c0984ff-509b-4910-8f46-437583c2f376/volumes" Oct 30 08:01:42.433088 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:42.433072 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" path="/var/lib/kubelet/pods/d3d9383d-2370-4553-bfb4-298acddff9d3/volumes" Oct 30 08:01:43.902281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:43.902246 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:45.626132 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:45.626082 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:45.877320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:45.877233 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 08:01:45.911533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:45.911496 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 08:01:46.030902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:46.030866 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:46.159756 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:46.157436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:47.355712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:47.355672 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:49.026269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:49.026222 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:50.637207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:50.637166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:50.878314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:50.878245 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" containerID="cri-o://9e77435235833b5ee1bc022b0773d5b266076f16fab866d6e387b1da436e7491" gracePeriod=25 Oct 30 08:01:50.912068 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:50.911974 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" containerID="cri-o://6e33d2f746580932fa261ad42dcc5a460c3716ad8b43cbb894587d184d132758" gracePeriod=25 Oct 30 08:01:51.801574 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:51.801531 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:52.332561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:52.332514 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:52.808806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:52.808769 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:01:54.075262 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:54.075218 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:55.817901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:55.817868 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:57.443029 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:57.442944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:58.316051 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316000 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98"] Oct 30 08:01:58.316753 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316734 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" Oct 30 08:01:58.316810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316754 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" Oct 30 08:01:58.316810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316764 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09754aae-6a10-4c29-ac00-a4ca7093633f" containerName="revoke-keys" Oct 30 08:01:58.316810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316770 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09754aae-6a10-4c29-ac00-a4ca7093633f" containerName="revoke-keys" Oct 30 08:01:58.316810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316793 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" Oct 30 08:01:58.316810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316799 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" Oct 30 08:01:58.316981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316891 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c0984ff-509b-4910-8f46-437583c2f376" containerName="payload-pre-processing" Oct 30 08:01:58.316981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316912 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d9383d-2370-4553-bfb4-298acddff9d3" containerName="payload-processing" Oct 30 08:01:58.316981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.316919 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09754aae-6a10-4c29-ac00-a4ca7093633f" containerName="revoke-keys" Oct 30 08:01:58.320613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.320593 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.340616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.340580 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98"] Oct 30 08:01:58.351593 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.351557 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:01:58.351593 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.351570 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:01:58.362449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.362413 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.362684 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.362549 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6s22\" (UniqueName: \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.362684 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.362591 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.463902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.463851 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.464443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.463989 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6s22\" (UniqueName: \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.464443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.464022 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.464679 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.464625 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.466567 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.466543 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.489769 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.489743 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6s22\" (UniqueName: \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\") pod \"maas-api-revoke-keys-e2e-extmod-a-705337-rkx98\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.635329 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.635241 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:01:58.951869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:58.951836 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98"] Oct 30 08:01:58.952407 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:01:58.952378 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb250d63a_6c7d_4250_95c0_558cfd7109c8.slice/crio-33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9 WatchSource:0}: Error finding container 33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9: Status 404 returned error can't find the container with id 33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9 Oct 30 08:01:59.161564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:59.161527 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:01:59.963536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:59.963501 5894 generic.go:356] "Generic (PLEG): container finished" podID="b250d63a-6c7d-4250-95c0-558cfd7109c8" containerID="b8f7d2a53d41005144292c07d4eb4e235c7bcc741cb93981466a852931f548ee" exitCode=0 Oct 30 08:01:59.963964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:59.963582 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" event={"ID":"b250d63a-6c7d-4250-95c0-558cfd7109c8","Type":"ContainerDied","Data":"b8f7d2a53d41005144292c07d4eb4e235c7bcc741cb93981466a852931f548ee"} Oct 30 08:01:59.963964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:01:59.963653 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" event={"ID":"b250d63a-6c7d-4250-95c0-558cfd7109c8","Type":"ContainerStarted","Data":"33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9"} Oct 30 08:02:00.797341 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:00.797291 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:01.192841 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.192812 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:02:01.305120 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.305029 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\") pod \"b250d63a-6c7d-4250-95c0-558cfd7109c8\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " Oct 30 08:02:01.305281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.305184 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\" (UniqueName: \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\") pod \"b250d63a-6c7d-4250-95c0-558cfd7109c8\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " Oct 30 08:02:01.305281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.305266 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\") pod \"b250d63a-6c7d-4250-95c0-558cfd7109c8\" (UID: \"b250d63a-6c7d-4250-95c0-558cfd7109c8\") " Oct 30 08:02:01.305712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.305680 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca" pod "b250d63a-6c7d-4250-95c0-558cfd7109c8" (UID: "b250d63a-6c7d-4250-95c0-558cfd7109c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:01.307330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.307299 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca" pod "b250d63a-6c7d-4250-95c0-558cfd7109c8" (UID: "b250d63a-6c7d-4250-95c0-558cfd7109c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:01.307470 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.307419 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22" pod "b250d63a-6c7d-4250-95c0-558cfd7109c8" (UID: "b250d63a-6c7d-4250-95c0-558cfd7109c8"). InnerVolumeSpecName "kube-api-access-x6s22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:01.406888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.406852 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6s22\" (UniqueName: \"kubernetes.io/projected/b250d63a-6c7d-4250-95c0-558cfd7109c8-kube-api-access-x6s22\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:01.406888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.406885 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:01.407144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.406898 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b250d63a-6c7d-4250-95c0-558cfd7109c8-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:01.801487 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.801443 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:01.991491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.991460 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" Oct 30 08:02:01.991491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.991479 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98" event={"ID":"b250d63a-6c7d-4250-95c0-558cfd7109c8","Type":"ContainerDied","Data":"33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9"} Oct 30 08:02:01.991763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:01.991533 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b25ced5b61472dad6750b20db6936d4bf4fe7daa5e18ce8d7a808c13f171d9" Oct 30 08:02:02.526290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:02.526255 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:02.808561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:02.808510 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:04.136942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.136904 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:04.216947 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.216911 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:02:04.217905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.217887 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b250d63a-6c7d-4250-95c0-558cfd7109c8" containerName="revoke-keys" Oct 30 08:02:04.217981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.217908 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b250d63a-6c7d-4250-95c0-558cfd7109c8" containerName="revoke-keys" Oct 30 08:02:04.218104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.218093 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b250d63a-6c7d-4250-95c0-558cfd7109c8" containerName="revoke-keys" Oct 30 08:02:04.222717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.222688 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.248201 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.248165 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:02:04.258184 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.258147 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"openshift-service-ca.crt\"" Oct 30 08:02:04.258184 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.258156 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"kube-root-ca.crt\"" Oct 30 08:02:04.258373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.258190 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"default-dockercfg-zlnwx\"" Oct 30 08:02:04.258373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.258327 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-d89593\"/\"rate-test-model-d89593-kserve-self-signed-certs\"" Oct 30 08:02:04.276816 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.276780 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:04.347423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.347372 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.347614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.347563 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.347735 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.347704 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.347854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.347825 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwldf\" (UniqueName: \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.347988 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.347971 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.348039 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.348011 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.453588 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.453701 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwldf\" (UniqueName: \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.453819 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.453858 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.453942 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.454054 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.454553 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.456712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.454847 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.461656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.458041 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.473656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.465981 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.498686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.489028 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.498686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.493363 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwldf\" (UniqueName: \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\") pod \"rate-test-model-d89593-kserve-5864855677-n7mgw\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.558257 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.557119 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:04.838282 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:02:04.838182 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:02:04.886764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:04.886732 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:02:04.886926 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:02:04.886898 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8178f323_24f3_42ad_be83_1fd8d0edd7d7.slice/crio-d84666487c793137edfdbc2af80316c694c85bc3302a8acd78bc54d9d21702b3 WatchSource:0}: Error finding container d84666487c793137edfdbc2af80316c694c85bc3302a8acd78bc54d9d21702b3: Status 404 returned error can't find the container with id d84666487c793137edfdbc2af80316c694c85bc3302a8acd78bc54d9d21702b3 Oct 30 08:02:05.020079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:05.020035 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" event={"ID":"8178f323-24f3-42ad-be83-1fd8d0edd7d7","Type":"ContainerStarted","Data":"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f"} Oct 30 08:02:05.020285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:05.020091 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" event={"ID":"8178f323-24f3-42ad-be83-1fd8d0edd7d7","Type":"ContainerStarted","Data":"d84666487c793137edfdbc2af80316c694c85bc3302a8acd78bc54d9d21702b3"} Oct 30 08:02:05.068145 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:05.068089 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" podStartSLOduration=1.068075085 podStartE2EDuration="1.068075085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:05.066367535 +0000 UTC m=+3888.757284973" watchObservedRunningTime="2026-10-30 08:02:05.068075085 +0000 UTC m=+3888.758992565" Oct 30 08:02:05.558935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:05.558900 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:05.927169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:05.927070 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:07.567956 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:07.567923 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:08.785944 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:08.785903 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 08:02:08.821650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:08.821595 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 08:02:08.999017 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:08.998975 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:09.129661 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:09.129560 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:09.263646 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:09.263611 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:09.782566 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:09.782529 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:11.023431 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.023400 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:11.104227 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.104193 5894 generic.go:356] "Generic (PLEG): container finished" podID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerID="9e77435235833b5ee1bc022b0773d5b266076f16fab866d6e387b1da436e7491" exitCode=0 Oct 30 08:02:11.104547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.104528 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" event={"ID":"09d33be6-7a14-4c77-b0d8-af05de28a805","Type":"ContainerDied","Data":"9e77435235833b5ee1bc022b0773d5b266076f16fab866d6e387b1da436e7491"} Oct 30 08:02:11.109554 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.109508 5894 generic.go:356] "Generic (PLEG): container finished" podID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerID="6e33d2f746580932fa261ad42dcc5a460c3716ad8b43cbb894587d184d132758" exitCode=0 Oct 30 08:02:11.109793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.109716 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" event={"ID":"c705bd40-a916-4f80-a2e7-416cc8f495d2","Type":"ContainerDied","Data":"6e33d2f746580932fa261ad42dcc5a460c3716ad8b43cbb894587d184d132758"} Oct 30 08:02:11.250354 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.250321 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 08:02:11.283245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.283219 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\" (UniqueName: \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\") pod \"09d33be6-7a14-4c77-b0d8-af05de28a805\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " Oct 30 08:02:11.283414 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.283386 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 08:02:11.283483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.283424 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\") pod \"09d33be6-7a14-4c77-b0d8-af05de28a805\" (UID: \"09d33be6-7a14-4c77-b0d8-af05de28a805\") " Oct 30 08:02:11.283828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.283797 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume" pod "09d33be6-7a14-4c77-b0d8-af05de28a805" (UID: "09d33be6-7a14-4c77-b0d8-af05de28a805"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:11.284145 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.284127 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09d33be6-7a14-4c77-b0d8-af05de28a805-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:11.285718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.285695 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq" pod "09d33be6-7a14-4c77-b0d8-af05de28a805" (UID: "09d33be6-7a14-4c77-b0d8-af05de28a805"). InnerVolumeSpecName "kube-api-access-2rntq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:11.385508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.385465 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\" (UniqueName: \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\") pod \"c705bd40-a916-4f80-a2e7-416cc8f495d2\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " Oct 30 08:02:11.385718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.385602 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\") pod \"c705bd40-a916-4f80-a2e7-416cc8f495d2\" (UID: \"c705bd40-a916-4f80-a2e7-416cc8f495d2\") " Oct 30 08:02:11.386232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.386202 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rntq\" (UniqueName: \"kubernetes.io/projected/09d33be6-7a14-4c77-b0d8-af05de28a805-kube-api-access-2rntq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:11.386735 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.386700 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume" pod "c705bd40-a916-4f80-a2e7-416cc8f495d2" (UID: "c705bd40-a916-4f80-a2e7-416cc8f495d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:11.388048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.388025 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f" pod "c705bd40-a916-4f80-a2e7-416cc8f495d2" (UID: "c705bd40-a916-4f80-a2e7-416cc8f495d2"). InnerVolumeSpecName "kube-api-access-zq56f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:11.487711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.487561 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq56f\" (UniqueName: \"kubernetes.io/projected/c705bd40-a916-4f80-a2e7-416cc8f495d2-kube-api-access-zq56f\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:11.487711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:11.487602 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c705bd40-a916-4f80-a2e7-416cc8f495d2-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:12.119876 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.119838 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" Oct 30 08:02:12.119876 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.119861 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8" event={"ID":"09d33be6-7a14-4c77-b0d8-af05de28a805","Type":"ContainerDied","Data":"4e1b1b22ca008c865e919abf9fe2f7a68df510ecca662874e961509b985b7760"} Oct 30 08:02:12.120433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.119929 5894 scope.go:122] "RemoveContainer" containerID="9e77435235833b5ee1bc022b0773d5b266076f16fab866d6e387b1da436e7491" Oct 30 08:02:12.121473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.121446 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" Oct 30 08:02:12.121473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.121458 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd" event={"ID":"c705bd40-a916-4f80-a2e7-416cc8f495d2","Type":"ContainerDied","Data":"6ca93d290741a184308785e66c7c08f9bbde68ca6c59d38bec03ad25b0d69e83"} Oct 30 08:02:12.132847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.132824 5894 scope.go:122] "RemoveContainer" containerID="6e33d2f746580932fa261ad42dcc5a460c3716ad8b43cbb894587d184d132758" Oct 30 08:02:12.179427 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.179336 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 08:02:12.193575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.193537 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-b3a7d7-845db69d54-shtz8"] Oct 30 08:02:12.221403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.221366 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 08:02:12.224230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.224206 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b3a7d7-787d495955-hd6wd"] Oct 30 08:02:12.433357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.433275 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" path="/var/lib/kubelet/pods/09d33be6-7a14-4c77-b0d8-af05de28a805/volumes" Oct 30 08:02:12.433764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.433747 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" path="/var/lib/kubelet/pods/c705bd40-a916-4f80-a2e7-416cc8f495d2/volumes" Oct 30 08:02:12.625907 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:12.625872 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:13.786344 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:13.786299 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" containerID="cri-o://eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba" gracePeriod=25 Oct 30 08:02:13.822807 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:13.822743 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" containerID="cri-o://d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435" gracePeriod=25 Oct 30 08:02:14.360255 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:14.360219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:14.559014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:14.558982 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:14.572584 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:14.572555 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:15.172009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:15.171982 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:02:15.224063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:15.224023 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:15.224246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:15.224083 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:16.042411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:16.042365 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:18.033057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:18.033016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:19.285045 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:19.285001 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:02:19.698183 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:19.698085 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:21.467336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:21.467301 5894 scope.go:122] "RemoveContainer" containerID="8c9931a7df0edf828a239e3bf5535cd7a46b53051bfdb3ba08ecb2151ce96d9a" Oct 30 08:02:21.482308 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:21.482280 5894 scope.go:122] "RemoveContainer" containerID="bc768bb438423670da121a99dcf2b81a341f4fa0fb429141eef3c4f1b5411f8e" Oct 30 08:02:21.498186 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:21.498160 5894 scope.go:122] "RemoveContainer" containerID="643b1fb67d07ba112640ea2f4ddb2c00228bb5bc26c1677b699ae74c25edeed8" Oct 30 08:02:21.513352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:21.513317 5894 scope.go:122] "RemoveContainer" containerID="2bebb0fbeb760ec89c9d885747b09faf52b91feef79ffbecd8f9a14cae9aa9e9" Oct 30 08:02:21.628525 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:21.628484 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:23.443760 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:23.443722 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:24.910345 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:24.910309 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:25.136213 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.136136 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:25.228716 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.226657 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:25.233615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.231711 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:02:25.623281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.623221 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:25.786459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.786415 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb"] Oct 30 08:02:25.787961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.787919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:25.797837 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:25.797788 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-b3a7d7-g75vb"] Oct 30 08:02:26.286656 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:26.286600 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:26.434095 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:26.434059 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09754aae-6a10-4c29-ac00-a4ca7093633f" path="/var/lib/kubelet/pods/09754aae-6a10-4c29-ac00-a4ca7093633f/volumes" Oct 30 08:02:27.599362 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:27.599322 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:29.256846 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:29.256807 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:31.044451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:31.044415 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:32.829106 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:32.829071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:33.322986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:33.322942 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:34.149448 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.149072 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 08:02:34.217244 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.217215 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 08:02:34.257024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.256989 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:34.289215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.289174 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\") pod \"83d4057f-3238-4f1a-96fb-b28345420717\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " Oct 30 08:02:34.289421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.289320 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\" (UniqueName: \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\") pod \"83d4057f-3238-4f1a-96fb-b28345420717\" (UID: \"83d4057f-3238-4f1a-96fb-b28345420717\") " Oct 30 08:02:34.289710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.289674 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume" pod "83d4057f-3238-4f1a-96fb-b28345420717" (UID: "83d4057f-3238-4f1a-96fb-b28345420717"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:34.290085 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.290066 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83d4057f-3238-4f1a-96fb-b28345420717-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:34.292482 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.292462 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9" pod "83d4057f-3238-4f1a-96fb-b28345420717" (UID: "83d4057f-3238-4f1a-96fb-b28345420717"). InnerVolumeSpecName "kube-api-access-wwfg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:34.347247 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.347138 5894 generic.go:356] "Generic (PLEG): container finished" podID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerID="d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435" exitCode=0 Oct 30 08:02:34.347247 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.347201 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" Oct 30 08:02:34.347517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.347243 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" event={"ID":"89a1b98c-1783-423c-9ed0-16b7ae07d9f5","Type":"ContainerDied","Data":"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435"} Oct 30 08:02:34.347517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.347302 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg" event={"ID":"89a1b98c-1783-423c-9ed0-16b7ae07d9f5","Type":"ContainerDied","Data":"dc25ca64def56d53111cdaa097fe6f700b514d8b8eaf745cefb175b2cbf69cb3"} Oct 30 08:02:34.347517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.347419 5894 scope.go:122] "RemoveContainer" containerID="d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435" Oct 30 08:02:34.349006 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.348973 5894 generic.go:356] "Generic (PLEG): container finished" podID="83d4057f-3238-4f1a-96fb-b28345420717" containerID="eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba" exitCode=0 Oct 30 08:02:34.349101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.349015 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" Oct 30 08:02:34.349212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.349195 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" event={"ID":"83d4057f-3238-4f1a-96fb-b28345420717","Type":"ContainerDied","Data":"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba"} Oct 30 08:02:34.349253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.349237 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr" event={"ID":"83d4057f-3238-4f1a-96fb-b28345420717","Type":"ContainerDied","Data":"8ec1c64457ca0ec2f9b40f73a56a153cf7d7530f08b04732738159dff524ee83"} Oct 30 08:02:34.359689 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.359629 5894 scope.go:122] "RemoveContainer" containerID="d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435" Oct 30 08:02:34.360000 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:02:34.359975 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435\": container with ID starting with d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435 not found: ID does not exist" containerID="d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435" Oct 30 08:02:34.360060 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.360008 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435"} err="failed to get container status \"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435\": rpc error: code = NotFound desc = could not find container \"d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435\": container with ID starting with d504b97819192a416ab9ccf9fd89787a802b57667ac863a8b8db8134e3d9b435 not found: ID does not exist" Oct 30 08:02:34.360060 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.360027 5894 scope.go:122] "RemoveContainer" containerID="eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba" Oct 30 08:02:34.370949 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.370929 5894 scope.go:122] "RemoveContainer" containerID="eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba" Oct 30 08:02:34.371922 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:02:34.371897 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba\": container with ID starting with eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba not found: ID does not exist" containerID="eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba" Oct 30 08:02:34.372008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.371932 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba"} err="failed to get container status \"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba\": rpc error: code = NotFound desc = could not find container \"eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba\": container with ID starting with eabe197793fb59bb1ae7f8a8afa2f0d47afd9a477e55de7c8a0ef2735e8d19ba not found: ID does not exist" Oct 30 08:02:34.377452 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.377412 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:34.391235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.391204 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\" (UniqueName: \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\") pod \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " Oct 30 08:02:34.391368 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.391240 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\") pod \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\" (UID: \"89a1b98c-1783-423c-9ed0-16b7ae07d9f5\") " Oct 30 08:02:34.391627 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.391613 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwfg9\" (UniqueName: \"kubernetes.io/projected/83d4057f-3238-4f1a-96fb-b28345420717-kube-api-access-wwfg9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:34.391627 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.391617 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume" pod "89a1b98c-1783-423c-9ed0-16b7ae07d9f5" (UID: "89a1b98c-1783-423c-9ed0-16b7ae07d9f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:34.393423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.393402 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg" pod "89a1b98c-1783-423c-9ed0-16b7ae07d9f5" (UID: "89a1b98c-1783-423c-9ed0-16b7ae07d9f5"). InnerVolumeSpecName "kube-api-access-6x8hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:34.418791 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.418756 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 08:02:34.422123 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.422092 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-705337-74cc7bb586-srmzr"] Oct 30 08:02:34.435022 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.434988 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83d4057f-3238-4f1a-96fb-b28345420717" path="/var/lib/kubelet/pods/83d4057f-3238-4f1a-96fb-b28345420717/volumes" Oct 30 08:02:34.493560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.493513 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x8hg\" (UniqueName: \"kubernetes.io/projected/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-kube-api-access-6x8hg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:34.493560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.493557 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89a1b98c-1783-423c-9ed0-16b7ae07d9f5-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:34.651758 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.651666 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:34.722872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.722497 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 08:02:34.752670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.742111 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-705337-66d76c4bc5-cnqgg"] Oct 30 08:02:34.822207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:34.818053 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:34.887687 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:02:34.887499 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:02:35.057088 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.057037 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:02:35.058204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058160 5894 cpu_manager.go:414] "containerMap: removing container" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" Oct 30 08:02:35.058204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058191 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058221 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058230 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058248 5894 cpu_manager.go:414] "containerMap: removing container" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058256 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058292 5894 cpu_manager.go:414] "containerMap: removing container" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" Oct 30 08:02:35.058398 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058300 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" Oct 30 08:02:35.058694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058480 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" containerName="payload-pre-processing" Oct 30 08:02:35.058694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058510 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="83d4057f-3238-4f1a-96fb-b28345420717" containerName="payload-processing" Oct 30 08:02:35.058694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058527 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c705bd40-a916-4f80-a2e7-416cc8f495d2" containerName="payload-pre-processing" Oct 30 08:02:35.058694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.058540 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d33be6-7a14-4c77-b0d8-af05de28a805" containerName="payload-processing" Oct 30 08:02:35.064684 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.064605 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.085226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.084589 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"default-dockercfg-76wzq\"" Oct 30 08:02:35.085226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.085046 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"rate-test-model-054ef1-kserve-self-signed-certs\"" Oct 30 08:02:35.087215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.087187 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:02:35.095960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.095926 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"openshift-service-ca.crt\"" Oct 30 08:02:35.096113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.095968 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-054ef1\"/\"kube-root-ca.crt\"" Oct 30 08:02:35.114658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.114601 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.114830 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.114762 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.114901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.114874 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.115006 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.114978 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.115077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.115027 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lvs9\" (UniqueName: \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.115077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.115069 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.136357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.136321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:35.216328 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216288 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216369 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216434 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216504 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216543 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lvs9\" (UniqueName: \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216584 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216751 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.216902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.216825 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.217287 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.217015 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.219194 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.219169 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.219927 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.219904 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.241544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.241505 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl"] Oct 30 08:02:35.247432 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.247399 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lvs9\" (UniqueName: \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\") pod \"rate-test-model-054ef1-kserve-7b746d874f-x72pr\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.247613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.247450 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.265569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.265529 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl"] Oct 30 08:02:35.346807 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.343759 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:02:35.346807 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.343925 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:02:35.400700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.400667 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:35.446253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.446215 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkdb\" (UniqueName: \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.446468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.446269 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.446468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.446362 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.548017 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.547843 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkdb\" (UniqueName: \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.548017 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.547916 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.548306 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.548102 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.549164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.549132 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.550875 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.550849 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.573799 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.573761 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkdb\" (UniqueName: \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.583629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.583599 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:35.741913 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:02:35.741874 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63b41062_701f_40ff_9c4d_1b28c624e7b6.slice/crio-21721b3ee29185abead82e4ebd49a8401a6ea11fecbdfdaa30be14b36ee8f6a6 WatchSource:0}: Error finding container 21721b3ee29185abead82e4ebd49a8401a6ea11fecbdfdaa30be14b36ee8f6a6: Status 404 returned error can't find the container with id 21721b3ee29185abead82e4ebd49a8401a6ea11fecbdfdaa30be14b36ee8f6a6 Oct 30 08:02:35.743988 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.743963 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:02:35.892035 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:02:35.891989 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e65536_dd7b_4c2e_ada3_127f1b976d0e.slice/crio-49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8 WatchSource:0}: Error finding container 49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8: Status 404 returned error can't find the container with id 49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8 Oct 30 08:02:35.892674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:35.892557 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl"] Oct 30 08:02:36.407708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.407667 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" event={"ID":"d3e65536-dd7b-4c2e-ada3-127f1b976d0e","Type":"ContainerStarted","Data":"0b73d5a3393e1a5239cdae10e06bf85df3dd76b6735b9076265762fef0cab0f0"} Oct 30 08:02:36.408236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.407730 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" event={"ID":"d3e65536-dd7b-4c2e-ada3-127f1b976d0e","Type":"ContainerStarted","Data":"49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8"} Oct 30 08:02:36.409554 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.409528 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" event={"ID":"63b41062-701f-40ff-9c4d-1b28c624e7b6","Type":"ContainerStarted","Data":"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8"} Oct 30 08:02:36.409728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.409560 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" event={"ID":"63b41062-701f-40ff-9c4d-1b28c624e7b6","Type":"ContainerStarted","Data":"21721b3ee29185abead82e4ebd49a8401a6ea11fecbdfdaa30be14b36ee8f6a6"} Oct 30 08:02:36.432280 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.432237 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a1b98c-1783-423c-9ed0-16b7ae07d9f5" path="/var/lib/kubelet/pods/89a1b98c-1783-423c-9ed0-16b7ae07d9f5/volumes" Oct 30 08:02:36.452741 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.452600 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" podStartSLOduration=1.452585722 podStartE2EDuration="1.452585722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:36.451707112 +0000 UTC m=+3920.142624549" watchObservedRunningTime="2026-10-30 08:02:36.452585722 +0000 UTC m=+3920.143503195" Oct 30 08:02:36.504793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.504719 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" podStartSLOduration=1.504699493 podStartE2EDuration="1.504699493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:02:36.502112577 +0000 UTC m=+3920.193030026" watchObservedRunningTime="2026-10-30 08:02:36.504699493 +0000 UTC m=+3920.195616934" Oct 30 08:02:36.734789 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:36.734483 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:37.401153 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:37.401118 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:37.418377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:37.418340 5894 generic.go:356] "Generic (PLEG): container finished" podID="d3e65536-dd7b-4c2e-ada3-127f1b976d0e" containerID="0b73d5a3393e1a5239cdae10e06bf85df3dd76b6735b9076265762fef0cab0f0" exitCode=0 Oct 30 08:02:37.418849 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:37.418431 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" event={"ID":"d3e65536-dd7b-4c2e-ada3-127f1b976d0e","Type":"ContainerDied","Data":"0b73d5a3393e1a5239cdae10e06bf85df3dd76b6735b9076265762fef0cab0f0"} Oct 30 08:02:38.391879 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.391846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:38.612823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.612795 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:38.693664 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.693554 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\") pod \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " Oct 30 08:02:38.693850 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.693793 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\") pod \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " Oct 30 08:02:38.693930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.693895 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\" (UniqueName: \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\") pod \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\" (UID: \"d3e65536-dd7b-4c2e-ada3-127f1b976d0e\") " Oct 30 08:02:38.694256 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.694227 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca" pod "d3e65536-dd7b-4c2e-ada3-127f1b976d0e" (UID: "d3e65536-dd7b-4c2e-ada3-127f1b976d0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:02:38.694383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.694363 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:38.695804 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.695784 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca" pod "d3e65536-dd7b-4c2e-ada3-127f1b976d0e" (UID: "d3e65536-dd7b-4c2e-ada3-127f1b976d0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:02:38.696553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.696538 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb" pod "d3e65536-dd7b-4c2e-ada3-127f1b976d0e" (UID: "d3e65536-dd7b-4c2e-ada3-127f1b976d0e"). InnerVolumeSpecName "kube-api-access-6jkdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:02:38.795269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.795226 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jkdb\" (UniqueName: \"kubernetes.io/projected/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-kube-api-access-6jkdb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:38.795269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:38.795262 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3e65536-dd7b-4c2e-ada3-127f1b976d0e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:02:39.443128 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:39.443090 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" event={"ID":"d3e65536-dd7b-4c2e-ada3-127f1b976d0e","Type":"ContainerDied","Data":"49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8"} Oct 30 08:02:39.443359 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:39.443151 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ced55f3932b9bf74e7a4fe1f6e5565d2ec3381278c10cac4957b9eaecdb6b8" Oct 30 08:02:39.443359 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:39.443104 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl" Oct 30 08:02:40.080569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:40.080470 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:41.747763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:41.747721 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:43.483009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:43.482976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:45.251573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.251527 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:45.401447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.401413 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:45.414874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.414843 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:45.501828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.501749 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:02:45.863374 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.863330 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:45.983488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:45.983219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:47.018520 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:47.018484 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:48.697819 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:48.697781 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:50.147575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.147535 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:50.749163 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.749125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:50.877329 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.877289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:50.889721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.889690 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98"] Oct 30 08:02:50.903359 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.903323 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-705337-rkx98"] Oct 30 08:02:50.999672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:50.999568 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:51.387201 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:51.387160 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:52.432336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:52.432287 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b250d63a-6c7d-4250-95c0-558cfd7109c8" path="/var/lib/kubelet/pods/b250d63a-6c7d-4250-95c0-558cfd7109c8/volumes" Oct 30 08:02:52.862851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:52.862811 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:54.580811 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:54.580774 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:56.240447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:56.240411 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:57.966560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:57.966521 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:58.486175 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:58.486134 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:58.613023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:58.612985 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:58.826935 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:58.826890 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 08:02:58.827188 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:58.827163 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" containerID="cri-o://001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007" gracePeriod=30 Oct 30 08:02:58.855813 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:58.855772 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:59.366914 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.366876 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn"] Oct 30 08:02:59.367605 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.367590 5894 cpu_manager.go:414] "containerMap: removing container" podUID="d3e65536-dd7b-4c2e-ada3-127f1b976d0e" containerName="revoke-keys" Oct 30 08:02:59.367681 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.367606 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e65536-dd7b-4c2e-ada3-127f1b976d0e" containerName="revoke-keys" Oct 30 08:02:59.367750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.367739 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e65536-dd7b-4c2e-ada3-127f1b976d0e" containerName="revoke-keys" Oct 30 08:02:59.371705 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.371683 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.404704 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.404668 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn"] Oct 30 08:02:59.415305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.415273 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.415489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.415315 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.415489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.415468 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6fw\" (UniqueName: \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.436381 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.436349 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:02:59.436653 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.436490 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:02:59.516491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.516444 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.516736 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.516518 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.516791 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.516752 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6fw\" (UniqueName: \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.517345 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.517318 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.518941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.518918 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.544222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.544188 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6fw\" (UniqueName: \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\") pod \"maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.687343 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.687256 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:02:59.847713 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.847672 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:02:59.995539 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:02:59.995492 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8f4456_3e37_4f4b_b27d_684cb7b5a3af.slice/crio-11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d WatchSource:0}: Error finding container 11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d: Status 404 returned error can't find the container with id 11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d Oct 30 08:02:59.996402 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:02:59.996370 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn"] Oct 30 08:03:00.681414 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:00.681316 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" event={"ID":"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af","Type":"ContainerStarted","Data":"d866978edde2346b2107ace3baa2d45d356e3bd9aad14946ec7b6b9de9a10764"} Oct 30 08:03:00.681414 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:00.681372 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" event={"ID":"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af","Type":"ContainerStarted","Data":"11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d"} Oct 30 08:03:00.726124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:00.726070 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" podStartSLOduration=1.72605399 podStartE2EDuration="1.72605399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:00.725042071 +0000 UTC m=+3944.415959508" watchObservedRunningTime="2026-10-30 08:03:00.72605399 +0000 UTC m=+3944.416971426" Oct 30 08:03:01.641365 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:01.641189 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:01.689985 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:01.689954 5894 generic.go:356] "Generic (PLEG): container finished" podID="1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" containerID="d866978edde2346b2107ace3baa2d45d356e3bd9aad14946ec7b6b9de9a10764" exitCode=0 Oct 30 08:03:01.690421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:01.690035 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" event={"ID":"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af","Type":"ContainerDied","Data":"d866978edde2346b2107ace3baa2d45d356e3bd9aad14946ec7b6b9de9a10764"} Oct 30 08:03:02.858992 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.858925 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:03:02.964429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.964336 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\" (UniqueName: \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\") pod \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " Oct 30 08:03:02.964615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.964502 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\") pod \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " Oct 30 08:03:02.964615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.964584 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\") pod \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\" (UID: \"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af\") " Oct 30 08:03:02.964872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.964847 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca" pod "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" (UID: "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:02.965182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.965161 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:02.966582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.966545 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw" pod "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" (UID: "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af"). InnerVolumeSpecName "kube-api-access-kx6fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:02.966715 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:02.966600 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca" pod "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" (UID: "1e8f4456-3e37-4f4b-b27d-684cb7b5a3af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:03.066041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.066004 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:03.066041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.066036 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6fw\" (UniqueName: \"kubernetes.io/projected/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af-kube-api-access-kx6fw\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:03.359431 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.359391 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:03.371651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.371602 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm"] Oct 30 08:03:03.372666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.372623 5894 cpu_manager.go:414] "containerMap: removing container" podUID="1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" containerName="revoke-keys" Oct 30 08:03:03.372740 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.372670 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" containerName="revoke-keys" Oct 30 08:03:03.372871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.372859 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" containerName="revoke-keys" Oct 30 08:03:03.377239 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.377215 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.421153 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.421108 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm"] Oct 30 08:03:03.471912 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.471873 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccj2z\" (UniqueName: \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.472094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.471924 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.472251 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.472227 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.573333 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.573293 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.573550 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.573386 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccj2z\" (UniqueName: \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.573550 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.573418 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.574068 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.574042 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.575866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.575836 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.598786 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.598755 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccj2z\" (UniqueName: \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\") pod \"maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:03.706314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.706223 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" Oct 30 08:03:03.706314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.706233 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn" event={"ID":"1e8f4456-3e37-4f4b-b27d-684cb7b5a3af","Type":"ContainerDied","Data":"11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d"} Oct 30 08:03:03.706314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.706292 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11eb2e806c995d046be00e96f65877028979d78f2516f722bdf82b92534d734d" Oct 30 08:03:03.732461 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:03.732429 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:04.042592 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.042501 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm"] Oct 30 08:03:04.198057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.198032 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 08:03:04.283816 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.283780 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.283850 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.283893 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.283997 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284090 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284109 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.284216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284146 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284206 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284237 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.284425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284342 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.284425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284353 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\") pod \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\" (UID: \"a307f70f-8e15-40b8-b8b4-659c38b70b4c\") " Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284628 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.284883 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.285046 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.285068 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.285082 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.285096 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.285401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.285365 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:04.287148 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.287119 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:03:04.288002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.287978 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:04.288002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.287989 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:03:04.288222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.288199 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf" pod "a307f70f-8e15-40b8-b8b4-659c38b70b4c" (UID: "a307f70f-8e15-40b8-b8b4-659c38b70b4c"). InnerVolumeSpecName "kube-api-access-nr9hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:04.386614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.386580 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.386614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.386611 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9hf\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-kube-api-access-nr9hf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.386614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.386623 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.386888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.386656 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.386888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.386667 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a307f70f-8e15-40b8-b8b4-659c38b70b4c-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:04.513628 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.513592 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:04.716677 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.716618 5894 generic.go:356] "Generic (PLEG): container finished" podID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerID="001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007" exitCode=0 Oct 30 08:03:04.716885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.716685 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" Oct 30 08:03:04.716885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.716706 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" event={"ID":"a307f70f-8e15-40b8-b8b4-659c38b70b4c","Type":"ContainerDied","Data":"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007"} Oct 30 08:03:04.716885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.716757 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh" event={"ID":"a307f70f-8e15-40b8-b8b4-659c38b70b4c","Type":"ContainerDied","Data":"f18e0aad5683628226153c007191c1df959974878fec7a4af3307f5f005ddb82"} Oct 30 08:03:04.716885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.716775 5894 scope.go:122] "RemoveContainer" containerID="001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007" Oct 30 08:03:04.718560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.718527 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" event={"ID":"a8346e49-a943-4aa6-9cd5-71f8be7559f2","Type":"ContainerStarted","Data":"e637328affa888f951628404fd6f621a499e7bffc8e32b181f70402c4af27a31"} Oct 30 08:03:04.718717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.718581 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" event={"ID":"a8346e49-a943-4aa6-9cd5-71f8be7559f2","Type":"ContainerStarted","Data":"0e9485b519995f980c212961ff6a08d3f00769a8c458253108e5cbf3a884b479"} Oct 30 08:03:04.728571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.728554 5894 scope.go:122] "RemoveContainer" containerID="001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007" Oct 30 08:03:04.728875 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:03:04.728850 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007\": container with ID starting with 001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007 not found: ID does not exist" containerID="001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007" Oct 30 08:03:04.728994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.728882 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007"} err="failed to get container status \"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007\": rpc error: code = NotFound desc = could not find container \"001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007\": container with ID starting with 001905f283dc028d9cd4423a008b16a2e336f27854d5a4f1625d8fe241d0c007 not found: ID does not exist" Oct 30 08:03:04.779477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.779442 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 08:03:04.792704 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.792673 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-705337-openshift-default-5ffc9fb7f7-bc6xh"] Oct 30 08:03:04.829967 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:03:04.829877 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:03:04.835247 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:04.835198 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" podStartSLOduration=1.8351839650000001 podStartE2EDuration="1.835183965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:04.833099098 +0000 UTC m=+3948.524016535" watchObservedRunningTime="2026-10-30 08:03:04.835183965 +0000 UTC m=+3948.526101398" Oct 30 08:03:05.203358 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:05.203319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:05.729159 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:05.729120 5894 generic.go:356] "Generic (PLEG): container finished" podID="a8346e49-a943-4aa6-9cd5-71f8be7559f2" containerID="e637328affa888f951628404fd6f621a499e7bffc8e32b181f70402c4af27a31" exitCode=0 Oct 30 08:03:05.729382 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:05.729188 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" event={"ID":"a8346e49-a943-4aa6-9cd5-71f8be7559f2","Type":"ContainerDied","Data":"e637328affa888f951628404fd6f621a499e7bffc8e32b181f70402c4af27a31"} Oct 30 08:03:06.433476 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.433441 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" path="/var/lib/kubelet/pods/a307f70f-8e15-40b8-b8b4-659c38b70b4c/volumes" Oct 30 08:03:06.895923 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.895890 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:06.920176 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.920136 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\") pod \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " Oct 30 08:03:06.920358 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.920255 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\" (UniqueName: \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\") pod \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " Oct 30 08:03:06.920449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.920378 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\") pod \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\" (UID: \"a8346e49-a943-4aa6-9cd5-71f8be7559f2\") " Oct 30 08:03:06.920793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.920768 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca" pod "a8346e49-a943-4aa6-9cd5-71f8be7559f2" (UID: "a8346e49-a943-4aa6-9cd5-71f8be7559f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:06.920936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.920914 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:06.922307 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.922282 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca" pod "a8346e49-a943-4aa6-9cd5-71f8be7559f2" (UID: "a8346e49-a943-4aa6-9cd5-71f8be7559f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:06.922486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:06.922462 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z" pod "a8346e49-a943-4aa6-9cd5-71f8be7559f2" (UID: "a8346e49-a943-4aa6-9cd5-71f8be7559f2"). InnerVolumeSpecName "kube-api-access-ccj2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:07.015582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.015486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:07.021882 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.021849 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8346e49-a943-4aa6-9cd5-71f8be7559f2-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:07.022008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.021901 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccj2z\" (UniqueName: \"kubernetes.io/projected/a8346e49-a943-4aa6-9cd5-71f8be7559f2-kube-api-access-ccj2z\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:07.746004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.745956 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" event={"ID":"a8346e49-a943-4aa6-9cd5-71f8be7559f2","Type":"ContainerDied","Data":"0e9485b519995f980c212961ff6a08d3f00769a8c458253108e5cbf3a884b479"} Oct 30 08:03:07.746516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.746017 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e9485b519995f980c212961ff6a08d3f00769a8c458253108e5cbf3a884b479" Oct 30 08:03:07.746516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:07.745972 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm" Oct 30 08:03:08.631468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:08.631431 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:09.840475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:09.840440 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 08:03:09.878065 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:09.878028 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 08:03:10.004237 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:10.004199 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:10.123146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:10.122890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:10.480384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:10.480289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:12.146595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:12.146555 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:13.524905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:13.524867 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:14.144536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.144500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:14.144992 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.144961 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:03:14.254389 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.254352 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl"] Oct 30 08:03:14.257193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.257162 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-54f2b2-xxvkl"] Oct 30 08:03:14.266158 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.266127 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:14.433239 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.433144 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e65536-dd7b-4c2e-ada3-127f1b976d0e" path="/var/lib/kubelet/pods/d3e65536-dd7b-4c2e-ada3-127f1b976d0e/volumes" Oct 30 08:03:14.841376 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.841331 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" containerID="cri-o://b145a75886ecbc86c81aa455a04fe271b065eff0b24df9d8111339adaea6958f" gracePeriod=25 Oct 30 08:03:14.878860 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.878808 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" containerID="cri-o://6c587bf9173acb24967658c393000c958ea3157574481a2376fecac8f0197e83" gracePeriod=25 Oct 30 08:03:14.917788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:14.917598 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:16.365829 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:16.365781 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:18.059800 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:18.059759 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:03:18.060229 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:18.059871 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:03:18.135693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:18.135659 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:19.411915 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.411876 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v"] Oct 30 08:03:19.412652 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412618 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" Oct 30 08:03:19.412652 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412653 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" Oct 30 08:03:19.412851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412685 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a8346e49-a943-4aa6-9cd5-71f8be7559f2" containerName="revoke-keys" Oct 30 08:03:19.412851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412691 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8346e49-a943-4aa6-9cd5-71f8be7559f2" containerName="revoke-keys" Oct 30 08:03:19.412851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412823 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8346e49-a943-4aa6-9cd5-71f8be7559f2" containerName="revoke-keys" Oct 30 08:03:19.412851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.412841 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a307f70f-8e15-40b8-b8b4-659c38b70b4c" containerName="istio-proxy" Oct 30 08:03:19.418019 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.417985 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.435241 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.435208 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v"] Oct 30 08:03:19.444924 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.444885 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:03:19.445080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.444953 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:03:19.613093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.613056 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsks\" (UniqueName: \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.615287 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.613118 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.615287 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.613301 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.715284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.715187 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.715464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.715321 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsks\" (UniqueName: \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.715464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.715347 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.715925 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.715897 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.717754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.717726 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.740909 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.740876 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsks\" (UniqueName: \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\") pod \"maas-api-revoke-sub-ac3f2a68-xrq4v\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:19.818356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:19.818318 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:20.034562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:20.034532 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:20.341959 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:20.341925 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v"] Oct 30 08:03:20.344922 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:20.344881 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffc6ad2_a607_4e99_8991_675bfd3a0ea3.slice/crio-6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea WatchSource:0}: Error finding container 6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea: Status 404 returned error can't find the container with id 6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea Oct 30 08:03:20.867109 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:20.867018 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" event={"ID":"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3","Type":"ContainerStarted","Data":"f93cbcd013989b411287dd8f8747f31f9b8202cd23f7de24b16fbd8b2bd81e8c"} Oct 30 08:03:20.867524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:20.867114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" event={"ID":"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3","Type":"ContainerStarted","Data":"6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea"} Oct 30 08:03:20.912126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:20.912071 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" podStartSLOduration=1.9120583450000002 podStartE2EDuration="1.912058345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:20.911219048 +0000 UTC m=+3964.602136486" watchObservedRunningTime="2026-10-30 08:03:20.912058345 +0000 UTC m=+3964.602975782" Oct 30 08:03:21.303457 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.303420 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:21.439477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.439446 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:21.633370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.633293 5894 scope.go:122] "RemoveContainer" containerID="a12697058a37a35ec2b46b6dcbe5afaa83b69727c4b21c3ed0376b7b514946e3" Oct 30 08:03:21.726887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.726854 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:21.890714 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.876363 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:21.908813 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.905162 5894 generic.go:356] "Generic (PLEG): container finished" podID="4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" containerID="f93cbcd013989b411287dd8f8747f31f9b8202cd23f7de24b16fbd8b2bd81e8c" exitCode=0 Oct 30 08:03:21.908813 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:21.905476 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" event={"ID":"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3","Type":"ContainerDied","Data":"f93cbcd013989b411287dd8f8747f31f9b8202cd23f7de24b16fbd8b2bd81e8c"} Oct 30 08:03:22.263019 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.262925 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj"] Oct 30 08:03:22.267013 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.266991 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.288155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.288122 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj"] Oct 30 08:03:22.349173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.349134 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.349361 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.349190 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.349361 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.349270 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4qz7\" (UniqueName: \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.450535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.450497 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.450763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.450554 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.450763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.450676 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4qz7\" (UniqueName: \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.451181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.451160 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.453044 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.453020 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.476023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.475993 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4qz7\" (UniqueName: \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.583953 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.583911 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:22.896865 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:22.896834 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a175bd3_f38f_4407_ba4d_727c4f3918a6.slice/crio-8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d WatchSource:0}: Error finding container 8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d: Status 404 returned error can't find the container with id 8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d Oct 30 08:03:22.898004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.897973 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj"] Oct 30 08:03:22.921196 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:22.921162 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" event={"ID":"6a175bd3-f38f-4407-ba4d-727c4f3918a6","Type":"ContainerStarted","Data":"8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d"} Oct 30 08:03:23.081889 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.081863 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:23.160202 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.160097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\") pod \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " Oct 30 08:03:23.160392 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.160220 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\") pod \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " Oct 30 08:03:23.160467 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.160401 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\" (UniqueName: \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\") pod \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\" (UID: \"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3\") " Oct 30 08:03:23.160624 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.160594 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca" pod "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" (UID: "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:23.161012 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.160985 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:23.163066 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.163039 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca" pod "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" (UID: "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:23.163066 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.163057 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks" pod "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" (UID: "4ffc6ad2-a607-4e99-8991-675bfd3a0ea3"). InnerVolumeSpecName "kube-api-access-bfsks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:23.262119 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.262079 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsks\" (UniqueName: \"kubernetes.io/projected/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-kube-api-access-bfsks\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:23.262119 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.262115 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:23.528911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.528871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:23.931281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.931250 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" Oct 30 08:03:23.931823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.931252 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v" event={"ID":"4ffc6ad2-a607-4e99-8991-675bfd3a0ea3","Type":"ContainerDied","Data":"6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea"} Oct 30 08:03:23.931823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.931393 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6da1caa5db3d7290799c1377fca75beeea644d8446929a9c68512eebf01865ea" Oct 30 08:03:23.933379 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.933348 5894 generic.go:356] "Generic (PLEG): container finished" podID="6a175bd3-f38f-4407-ba4d-727c4f3918a6" containerID="cfcc32de6a21acd8ed865294d2bcc9241c69d448b5620e9767aaaa6ac142a5fb" exitCode=0 Oct 30 08:03:23.933537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:23.933439 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" event={"ID":"6a175bd3-f38f-4407-ba4d-727c4f3918a6","Type":"ContainerDied","Data":"cfcc32de6a21acd8ed865294d2bcc9241c69d448b5620e9767aaaa6ac142a5fb"} Oct 30 08:03:25.120711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.120673 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:25.194309 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.194223 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\" (UniqueName: \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\") pod \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " Oct 30 08:03:25.194309 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.194289 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\") pod \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " Oct 30 08:03:25.194562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.194376 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\") pod \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\" (UID: \"6a175bd3-f38f-4407-ba4d-727c4f3918a6\") " Oct 30 08:03:25.194796 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.194768 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca" pod "6a175bd3-f38f-4407-ba4d-727c4f3918a6" (UID: "6a175bd3-f38f-4407-ba4d-727c4f3918a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:25.195070 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.195050 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:25.196512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.196479 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7" pod "6a175bd3-f38f-4407-ba4d-727c4f3918a6" (UID: "6a175bd3-f38f-4407-ba4d-727c4f3918a6"). InnerVolumeSpecName "kube-api-access-t4qz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:25.196608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.196576 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca" pod "6a175bd3-f38f-4407-ba4d-727c4f3918a6" (UID: "6a175bd3-f38f-4407-ba4d-727c4f3918a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:25.297282 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.297245 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a175bd3-f38f-4407-ba4d-727c4f3918a6-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:25.297282 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.297294 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4qz7\" (UniqueName: \"kubernetes.io/projected/6a175bd3-f38f-4407-ba4d-727c4f3918a6-kube-api-access-t4qz7\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:25.434539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.434486 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:25.951232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.951201 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" Oct 30 08:03:25.951420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.951232 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj" event={"ID":"6a175bd3-f38f-4407-ba4d-727c4f3918a6","Type":"ContainerDied","Data":"8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d"} Oct 30 08:03:25.951420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:25.951288 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e237c24242cfc3be32f4efd4df4f85486ff60d0e47d7e54a38b3ac89b56db1d" Oct 30 08:03:27.048169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:27.048087 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9"] Oct 30 08:03:27.061254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:27.061214 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c205912-qfhq9"] Oct 30 08:03:27.181610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:27.181569 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:28.044233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.044200 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb"] Oct 30 08:03:28.057481 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.057451 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ada2ab1d-xjmbb"] Oct 30 08:03:28.059463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.059422 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:03:28.059669 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.059477 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:03:28.433009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.432927 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2734439e-f511-42e3-848a-bdc8101e2563" path="/var/lib/kubelet/pods/2734439e-f511-42e3-848a-bdc8101e2563/volumes" Oct 30 08:03:28.433435 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.433420 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73c5261-5f2c-4b12-a2e1-50892e03ad54" path="/var/lib/kubelet/pods/c73c5261-5f2c-4b12-a2e1-50892e03ad54/volumes" Oct 30 08:03:28.805560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:28.805513 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:29.816075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816037 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf"] Oct 30 08:03:29.816698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816684 5894 cpu_manager.go:414] "containerMap: removing container" podUID="4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" containerName="revoke-keys" Oct 30 08:03:29.816768 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816700 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" containerName="revoke-keys" Oct 30 08:03:29.816768 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816726 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6a175bd3-f38f-4407-ba4d-727c4f3918a6" containerName="revoke-keys" Oct 30 08:03:29.816768 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816731 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a175bd3-f38f-4407-ba4d-727c4f3918a6" containerName="revoke-keys" Oct 30 08:03:29.816878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816822 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" containerName="revoke-keys" Oct 30 08:03:29.816878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.816833 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a175bd3-f38f-4407-ba4d-727c4f3918a6" containerName="revoke-keys" Oct 30 08:03:29.821279 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.821262 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:29.852809 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.852775 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:03:29.853002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.852826 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:03:29.856905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.856865 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf"] Oct 30 08:03:29.958842 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.958810 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:29.959042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.959023 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htw64\" (UniqueName: \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:29.959096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:29.959058 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.062195 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.062151 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htw64\" (UniqueName: \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.062479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.062457 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.062752 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.062732 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.063863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.063839 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.075052 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.074954 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.089988 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.089959 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htw64\" (UniqueName: \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\") pod \"maas-api-revoke-sub-0e3e0bb0-8svcf\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.135480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.135446 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:30.480600 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:30.480560 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0703ff75_2966_42c9_ae19_670b4cdc810c.slice/crio-8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb WatchSource:0}: Error finding container 8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb: Status 404 returned error can't find the container with id 8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb Oct 30 08:03:30.481521 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.481487 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf"] Oct 30 08:03:30.637317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.637281 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:30.988979 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.988937 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" event={"ID":"0703ff75-2966-42c9-ae19-670b4cdc810c","Type":"ContainerStarted","Data":"1af9582afb4c5ce00b42a1ff91ec9f94d84ed5ec0c814923a6c6022f259924f4"} Oct 30 08:03:30.989501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:30.989000 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" event={"ID":"0703ff75-2966-42c9-ae19-670b4cdc810c","Type":"ContainerStarted","Data":"8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb"} Oct 30 08:03:31.035355 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:31.035303 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" podStartSLOduration=2.035288345 podStartE2EDuration="2.035288345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:31.033189575 +0000 UTC m=+3974.724107012" watchObservedRunningTime="2026-10-30 08:03:31.035288345 +0000 UTC m=+3974.726205782" Oct 30 08:03:31.998009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:31.997977 5894 generic.go:356] "Generic (PLEG): container finished" podID="0703ff75-2966-42c9-ae19-670b4cdc810c" containerID="1af9582afb4c5ce00b42a1ff91ec9f94d84ed5ec0c814923a6c6022f259924f4" exitCode=0 Oct 30 08:03:31.998436 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:31.998035 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" event={"ID":"0703ff75-2966-42c9-ae19-670b4cdc810c","Type":"ContainerDied","Data":"1af9582afb4c5ce00b42a1ff91ec9f94d84ed5ec0c814923a6c6022f259924f4"} Oct 30 08:03:32.351130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:32.351097 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:32.958569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:32.958534 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:33.084843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.084805 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:33.218587 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.218520 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:33.328710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.328671 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\" (UniqueName: \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\") pod \"0703ff75-2966-42c9-ae19-670b4cdc810c\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " Oct 30 08:03:33.328870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.328842 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\") pod \"0703ff75-2966-42c9-ae19-670b4cdc810c\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " Oct 30 08:03:33.328938 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.328886 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\") pod \"0703ff75-2966-42c9-ae19-670b4cdc810c\" (UID: \"0703ff75-2966-42c9-ae19-670b4cdc810c\") " Oct 30 08:03:33.329260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.329230 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca" pod "0703ff75-2966-42c9-ae19-670b4cdc810c" (UID: "0703ff75-2966-42c9-ae19-670b4cdc810c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:33.329562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.329542 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:33.331053 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.331026 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca" pod "0703ff75-2966-42c9-ae19-670b4cdc810c" (UID: "0703ff75-2966-42c9-ae19-670b4cdc810c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:33.331197 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.331176 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64" pod "0703ff75-2966-42c9-ae19-670b4cdc810c" (UID: "0703ff75-2966-42c9-ae19-670b4cdc810c"). InnerVolumeSpecName "kube-api-access-htw64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:33.404600 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.404562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:33.441491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.439301 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htw64\" (UniqueName: \"kubernetes.io/projected/0703ff75-2966-42c9-ae19-670b4cdc810c-kube-api-access-htw64\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:33.441491 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:33.439338 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0703ff75-2966-42c9-ae19-670b4cdc810c-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:34.051826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:34.051792 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" event={"ID":"0703ff75-2966-42c9-ae19-670b4cdc810c","Type":"ContainerDied","Data":"8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb"} Oct 30 08:03:34.051991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:34.051843 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a5984687f3731af7fba0ce6fee1fcff90818401bf8a9cbb838958c3da09f5fb" Oct 30 08:03:34.051991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:34.051802 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf" Oct 30 08:03:34.101883 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:34.101846 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:34.828799 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:03:34.828752 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:03:35.070556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.070523 5894 generic.go:356] "Generic (PLEG): container finished" podID="0c381e5d-6156-45da-8738-ace575699213" containerID="b145a75886ecbc86c81aa455a04fe271b065eff0b24df9d8111339adaea6958f" exitCode=0 Oct 30 08:03:35.070767 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.070711 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" event={"ID":"0c381e5d-6156-45da-8738-ace575699213","Type":"ContainerDied","Data":"b145a75886ecbc86c81aa455a04fe271b065eff0b24df9d8111339adaea6958f"} Oct 30 08:03:35.072709 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.072682 5894 generic.go:356] "Generic (PLEG): container finished" podID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerID="6c587bf9173acb24967658c393000c958ea3157574481a2376fecac8f0197e83" exitCode=0 Oct 30 08:03:35.072946 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.072931 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" event={"ID":"fa177d1a-ab03-43f9-acb4-043698ffb066","Type":"ContainerDied","Data":"6c587bf9173acb24967658c393000c958ea3157574481a2376fecac8f0197e83"} Oct 30 08:03:35.135664 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.132509 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 08:03:35.200560 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.200527 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 08:03:35.267764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.267733 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\") pod \"0c381e5d-6156-45da-8738-ace575699213\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " Oct 30 08:03:35.267930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.267905 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\" (UniqueName: \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\") pod \"0c381e5d-6156-45da-8738-ace575699213\" (UID: \"0c381e5d-6156-45da-8738-ace575699213\") " Oct 30 08:03:35.268191 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.268162 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume" pod "0c381e5d-6156-45da-8738-ace575699213" (UID: "0c381e5d-6156-45da-8738-ace575699213"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:35.268425 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.268404 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c381e5d-6156-45da-8738-ace575699213-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.270263 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.270236 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w" pod "0c381e5d-6156-45da-8738-ace575699213" (UID: "0c381e5d-6156-45da-8738-ace575699213"). InnerVolumeSpecName "kube-api-access-xtk4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:35.369280 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.369191 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\") pod \"fa177d1a-ab03-43f9-acb4-043698ffb066\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " Oct 30 08:03:35.369453 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.369325 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\" (UniqueName: \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\") pod \"fa177d1a-ab03-43f9-acb4-043698ffb066\" (UID: \"fa177d1a-ab03-43f9-acb4-043698ffb066\") " Oct 30 08:03:35.369608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.369575 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume" pod "fa177d1a-ab03-43f9-acb4-043698ffb066" (UID: "fa177d1a-ab03-43f9-acb4-043698ffb066"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:35.369958 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.369935 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtk4w\" (UniqueName: \"kubernetes.io/projected/0c381e5d-6156-45da-8738-ace575699213-kube-api-access-xtk4w\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.369958 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.369956 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa177d1a-ab03-43f9-acb4-043698ffb066-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.371706 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.371683 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll" pod "fa177d1a-ab03-43f9-acb4-043698ffb066" (UID: "fa177d1a-ab03-43f9-acb4-043698ffb066"). InnerVolumeSpecName "kube-api-access-v9rll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:35.471523 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.471479 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rll\" (UniqueName: \"kubernetes.io/projected/fa177d1a-ab03-43f9-acb4-043698ffb066-kube-api-access-v9rll\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.472263 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.472232 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:03:35.472500 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.472475 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" podUID="55af87cf-850b-4613-b22c-471d18938461" containerName="maas-api" containerID="cri-o://a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b" gracePeriod=30 Oct 30 08:03:35.548616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.548553 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:35.785104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.780151 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:03:35.877240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.877205 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") pod \"55af87cf-850b-4613-b22c-471d18938461\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " Oct 30 08:03:35.877443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.877272 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\") pod \"55af87cf-850b-4613-b22c-471d18938461\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " Oct 30 08:03:35.877443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.877336 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\") pod \"55af87cf-850b-4613-b22c-471d18938461\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " Oct 30 08:03:35.877443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.877388 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\" (UniqueName: \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\") pod \"55af87cf-850b-4613-b22c-471d18938461\" (UID: \"55af87cf-850b-4613-b22c-471d18938461\") " Oct 30 08:03:35.878180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.878123 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca" pod "55af87cf-850b-4613-b22c-471d18938461" (UID: "55af87cf-850b-4613-b22c-471d18938461"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:35.880043 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.880019 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs" pod "55af87cf-850b-4613-b22c-471d18938461" (UID: "55af87cf-850b-4613-b22c-471d18938461"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:35.880169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.880144 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr" pod "55af87cf-850b-4613-b22c-471d18938461" (UID: "55af87cf-850b-4613-b22c-471d18938461"). InnerVolumeSpecName "kube-api-access-tcckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:35.880351 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.880332 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls" pod "55af87cf-850b-4613-b22c-471d18938461" (UID: "55af87cf-850b-4613-b22c-471d18938461"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:35.986773 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.983393 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-maas-api-tls\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.986773 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.983453 5894 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55af87cf-850b-4613-b22c-471d18938461-openshift-service-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.986773 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.983473 5894 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55af87cf-850b-4613-b22c-471d18938461-metrics-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:35.986773 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:35.983492 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcckr\" (UniqueName: \"kubernetes.io/projected/55af87cf-850b-4613-b22c-471d18938461-kube-api-access-tcckr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:36.083832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.083798 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" Oct 30 08:03:36.083832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.083823 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j" event={"ID":"fa177d1a-ab03-43f9-acb4-043698ffb066","Type":"ContainerDied","Data":"2b10f3ac52473f7dbb7e4b27f2e0f6a27e5ce7bf7d58511fb803ca747e8766c9"} Oct 30 08:03:36.084108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.083889 5894 scope.go:122] "RemoveContainer" containerID="6c587bf9173acb24967658c393000c958ea3157574481a2376fecac8f0197e83" Oct 30 08:03:36.085451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.085234 5894 generic.go:356] "Generic (PLEG): container finished" podID="55af87cf-850b-4613-b22c-471d18938461" containerID="a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b" exitCode=0 Oct 30 08:03:36.085451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.085284 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" Oct 30 08:03:36.085451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.085325 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" event={"ID":"55af87cf-850b-4613-b22c-471d18938461","Type":"ContainerDied","Data":"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b"} Oct 30 08:03:36.085451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.085434 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj" event={"ID":"55af87cf-850b-4613-b22c-471d18938461","Type":"ContainerDied","Data":"49ad464be3c993bfc941dac3d3756cee28657936dc52f28b087e816565c8db07"} Oct 30 08:03:36.087934 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.087882 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" event={"ID":"0c381e5d-6156-45da-8738-ace575699213","Type":"ContainerDied","Data":"a4f8f33d0f464d9a6e6b3c3c3f9356a9371157bbc31b6f4dc54e40a20baa9226"} Oct 30 08:03:36.088152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.087987 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm" Oct 30 08:03:36.098828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.098803 5894 scope.go:122] "RemoveContainer" containerID="a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b" Oct 30 08:03:36.102168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.102137 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:36.110722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.110693 5894 scope.go:122] "RemoveContainer" containerID="a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b" Oct 30 08:03:36.111136 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:03:36.111056 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b\": container with ID starting with a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b not found: ID does not exist" containerID="a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b" Oct 30 08:03:36.111136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.111085 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b"} err="failed to get container status \"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b\": rpc error: code = NotFound desc = could not find container \"a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b\": container with ID starting with a6ae614c0b68f047ea953f1c703ff6b6792bb912a316d9da279264d33c29db6b not found: ID does not exist" Oct 30 08:03:36.111136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.111104 5894 scope.go:122] "RemoveContainer" containerID="b145a75886ecbc86c81aa455a04fe271b065eff0b24df9d8111339adaea6958f" Oct 30 08:03:36.194480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.194446 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 08:03:36.198577 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.198546 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bffe43-c5b5d4759-scv6j"] Oct 30 08:03:36.223994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.223935 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:36.239082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.238979 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:03:36.245029 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.244992 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm"] Oct 30 08:03:36.248007 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.247980 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2d8539a1-5945cf8745-tbglj"] Oct 30 08:03:36.250584 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.250555 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2d8539a1-29zwm"] Oct 30 08:03:36.296623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.296577 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 08:03:36.302703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.302666 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bffe43-7779874858-fzfrm"] Oct 30 08:03:36.341596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.341562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:36.432038 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.432001 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c381e5d-6156-45da-8738-ace575699213" path="/var/lib/kubelet/pods/0c381e5d-6156-45da-8738-ace575699213/volumes" Oct 30 08:03:36.432558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.432535 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55af87cf-850b-4613-b22c-471d18938461" path="/var/lib/kubelet/pods/55af87cf-850b-4613-b22c-471d18938461/volumes" Oct 30 08:03:36.433052 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.433032 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8346e49-a943-4aa6-9cd5-71f8be7559f2" path="/var/lib/kubelet/pods/a8346e49-a943-4aa6-9cd5-71f8be7559f2/volumes" Oct 30 08:03:36.433824 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.433803 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" path="/var/lib/kubelet/pods/fa177d1a-ab03-43f9-acb4-043698ffb066/volumes" Oct 30 08:03:36.725083 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:36.725050 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:38.128595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:38.128561 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:39.041291 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:39.041252 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz"] Oct 30 08:03:39.054954 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:39.054922 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c32033fe-vdsqz"] Oct 30 08:03:39.788081 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:39.788045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:40.432128 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.432089 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7c7cc6e-26d2-42f6-8b54-489b6f0693c4" path="/var/lib/kubelet/pods/a7c7cc6e-26d2-42f6-8b54-489b6f0693c4/volumes" Oct 30 08:03:40.818138 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818106 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd"] Oct 30 08:03:40.818770 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818751 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818770 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818798 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0703ff75-2966-42c9-ae19-670b4cdc810c" containerName="revoke-keys" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818804 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0703ff75-2966-42c9-ae19-670b4cdc810c" containerName="revoke-keys" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818823 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818828 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818852 5894 cpu_manager.go:414] "containerMap: removing container" podUID="55af87cf-850b-4613-b22c-471d18938461" containerName="maas-api" Oct 30 08:03:40.818877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818857 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55af87cf-850b-4613-b22c-471d18938461" containerName="maas-api" Oct 30 08:03:40.819137 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818941 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="55af87cf-850b-4613-b22c-471d18938461" containerName="maas-api" Oct 30 08:03:40.819137 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818953 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa177d1a-ab03-43f9-acb4-043698ffb066" containerName="payload-pre-processing" Oct 30 08:03:40.819137 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818962 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0703ff75-2966-42c9-ae19-670b4cdc810c" containerName="revoke-keys" Oct 30 08:03:40.819137 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.818970 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c381e5d-6156-45da-8738-ace575699213" containerName="payload-processing" Oct 30 08:03:40.824130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.824107 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:40.844235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.844203 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd"] Oct 30 08:03:40.855693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.855668 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:03:40.855844 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.855751 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:03:40.895976 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.895943 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:03:40.896148 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.896026 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:03:40.945575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.945540 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:40.945775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.945704 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:40.945775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:40.945748 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlsxn\" (UniqueName: \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.047534 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.047501 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.047751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.047554 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlsxn\" (UniqueName: \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.047751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.047717 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.048261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.048236 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.050098 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.050077 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.073044 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.072964 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlsxn\" (UniqueName: \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\") pod \"maas-api-revoke-sub-eaf4b0f5-7cfmd\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.138273 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.138233 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:41.194881 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.194838 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:41.485293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.485256 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd"] Oct 30 08:03:41.772290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.772198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:41.858036 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.857996 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn"] Oct 30 08:03:41.871804 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.871761 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bffe43-b62dn"] Oct 30 08:03:41.889553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:41.889518 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:42.165858 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:42.165819 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" event={"ID":"c61ef24b-724b-41b8-acb3-130099bcdb01","Type":"ContainerStarted","Data":"99c70fc11a000453d6a4c7237b267c01ebd202e90ee5df3efed486cf78adddc4"} Oct 30 08:03:42.166072 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:42.165883 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" event={"ID":"c61ef24b-724b-41b8-acb3-130099bcdb01","Type":"ContainerStarted","Data":"a7af8838e2c10bd2f7622360202f888b20ec02c8221238007ab416c48fd372f1"} Oct 30 08:03:42.210193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:42.210144 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" podStartSLOduration=2.210131565 podStartE2EDuration="2.210131565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:42.209573073 +0000 UTC m=+3985.900490509" watchObservedRunningTime="2026-10-30 08:03:42.210131565 +0000 UTC m=+3985.901049003" Oct 30 08:03:42.311224 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:42.311179 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:42.432766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:42.432732 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8f4456-3e37-4f4b-b27d-684cb7b5a3af" path="/var/lib/kubelet/pods/1e8f4456-3e37-4f4b-b27d-684cb7b5a3af/volumes" Oct 30 08:03:43.030228 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.030191 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl"] Oct 30 08:03:43.044651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.044598 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b65133b-m76xl"] Oct 30 08:03:43.175209 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.175171 5894 generic.go:356] "Generic (PLEG): container finished" podID="c61ef24b-724b-41b8-acb3-130099bcdb01" containerID="99c70fc11a000453d6a4c7237b267c01ebd202e90ee5df3efed486cf78adddc4" exitCode=0 Oct 30 08:03:43.175370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.175269 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" event={"ID":"c61ef24b-724b-41b8-acb3-130099bcdb01","Type":"ContainerDied","Data":"99c70fc11a000453d6a4c7237b267c01ebd202e90ee5df3efed486cf78adddc4"} Oct 30 08:03:43.739645 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.739609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:43.967903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:43.967864 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:44.159540 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.159501 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:44.365114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.365078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:44.381251 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.381223 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:44.444832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.444745 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa85c94e-1f58-4505-bd3a-d6d31c3ecb67" path="/var/lib/kubelet/pods/fa85c94e-1f58-4505-bd3a-d6d31c3ecb67/volumes" Oct 30 08:03:44.494887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.494846 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\") pod \"c61ef24b-724b-41b8-acb3-130099bcdb01\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " Oct 30 08:03:44.495155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.494999 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\" (UniqueName: \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\") pod \"c61ef24b-724b-41b8-acb3-130099bcdb01\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " Oct 30 08:03:44.495267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.495173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\") pod \"c61ef24b-724b-41b8-acb3-130099bcdb01\" (UID: \"c61ef24b-724b-41b8-acb3-130099bcdb01\") " Oct 30 08:03:44.496094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.496063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca" pod "c61ef24b-724b-41b8-acb3-130099bcdb01" (UID: "c61ef24b-724b-41b8-acb3-130099bcdb01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:44.511324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.511269 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn" pod "c61ef24b-724b-41b8-acb3-130099bcdb01" (UID: "c61ef24b-724b-41b8-acb3-130099bcdb01"). InnerVolumeSpecName "kube-api-access-wlsxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:44.517974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.517874 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca" pod "c61ef24b-724b-41b8-acb3-130099bcdb01" (UID: "c61ef24b-724b-41b8-acb3-130099bcdb01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:44.596691 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.596651 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:44.596691 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.596686 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61ef24b-724b-41b8-acb3-130099bcdb01-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:44.596691 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:44.596698 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlsxn\" (UniqueName: \"kubernetes.io/projected/c61ef24b-724b-41b8-acb3-130099bcdb01-kube-api-access-wlsxn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:45.192040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:45.191998 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" Oct 30 08:03:45.192480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:45.192042 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd" event={"ID":"c61ef24b-724b-41b8-acb3-130099bcdb01","Type":"ContainerDied","Data":"a7af8838e2c10bd2f7622360202f888b20ec02c8221238007ab416c48fd372f1"} Oct 30 08:03:45.192480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:45.192101 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7af8838e2c10bd2f7622360202f888b20ec02c8221238007ab416c48fd372f1" Oct 30 08:03:45.629154 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:45.629119 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:47.123075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.123041 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:03:47.123690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.123674 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c61ef24b-724b-41b8-acb3-130099bcdb01" containerName="revoke-keys" Oct 30 08:03:47.123690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.123689 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61ef24b-724b-41b8-acb3-130099bcdb01" containerName="revoke-keys" Oct 30 08:03:47.123871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.123793 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61ef24b-724b-41b8-acb3-130099bcdb01" containerName="revoke-keys" Oct 30 08:03:47.127551 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.127529 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.148868 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.148841 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:03:47.158990 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.158961 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"openshift-service-ca.crt\"" Oct 30 08:03:47.159166 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.159003 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"kube-root-ca.crt\"" Oct 30 08:03:47.159274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.159003 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"default-dockercfg-4f77b\"" Oct 30 08:03:47.159349 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.159055 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-2e8b2a\"/\"rate-test-model-2e8b2a-kserve-self-signed-certs\"" Oct 30 08:03:47.177441 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.177398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:47.231813 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.231775 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.232037 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.231911 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.232037 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.231956 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.232037 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.231982 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9csh\" (UniqueName: \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.232226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.232063 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.232226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.232142 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333153 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333109 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333364 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333218 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333364 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333348 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333382 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333416 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9csh\" (UniqueName: \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333516 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333470 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333544 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333619 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.333868 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.333768 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.335904 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.335880 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.336360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.336333 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.355475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.355414 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:47.360012 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.359985 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9csh\" (UniqueName: \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\") pod \"rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.445068 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.444977 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:47.841203 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:47.841162 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3093fbfe_5c48_49a7_b8e8_78bee4a6cc07.slice/crio-54d841099987873a547fe92dc5c299e13e17338b254290e4f0fb8a19debd4cbd WatchSource:0}: Error finding container 54d841099987873a547fe92dc5c299e13e17338b254290e4f0fb8a19debd4cbd: Status 404 returned error can't find the container with id 54d841099987873a547fe92dc5c299e13e17338b254290e4f0fb8a19debd4cbd Oct 30 08:03:47.842360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:47.842336 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:03:48.039027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.038988 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd"] Oct 30 08:03:48.045166 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.045133 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bc1de02-p74zd"] Oct 30 08:03:48.231486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.231397 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" event={"ID":"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07","Type":"ContainerStarted","Data":"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50"} Oct 30 08:03:48.231486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.231446 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" event={"ID":"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07","Type":"ContainerStarted","Data":"54d841099987873a547fe92dc5c299e13e17338b254290e4f0fb8a19debd4cbd"} Oct 30 08:03:48.281853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.281798 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" podStartSLOduration=1.281783293 podStartE2EDuration="1.281783293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:48.278846135 +0000 UTC m=+3991.969763572" watchObservedRunningTime="2026-10-30 08:03:48.281783293 +0000 UTC m=+3991.972700773" Oct 30 08:03:48.455472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.455434 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="306f0d60-f2c5-49d3-9cea-8ea663a6857a" path="/var/lib/kubelet/pods/306f0d60-f2c5-49d3-9cea-8ea663a6857a/volumes" Oct 30 08:03:48.455981 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:48.455886 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:49.035038 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.034998 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:49.164102 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.164067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:49.283840 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.283810 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:49.411478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.411393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:49.887689 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.887620 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6"] Oct 30 08:03:49.892200 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.892178 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:49.915421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.915377 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6"] Oct 30 08:03:49.925172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.925140 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:03:49.925345 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.925312 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:03:49.925421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.925392 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:03:49.925510 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.925493 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:03:49.972321 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.972278 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:49.972499 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.972473 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:49.972555 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:49.972540 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf8qk\" (UniqueName: \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.073805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.073758 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.074020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.073824 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf8qk\" (UniqueName: \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.074105 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.074023 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.074508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.074473 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.076588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.076565 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.099096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.099067 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf8qk\" (UniqueName: \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.208537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.208455 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:50.525855 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:50.525824 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda464898b_d09d_4bb1_a02d_572875eca6ef.slice/crio-04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013 WatchSource:0}: Error finding container 04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013: Status 404 returned error can't find the container with id 04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013 Oct 30 08:03:50.526326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:50.526196 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6"] Oct 30 08:03:51.150238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.150152 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:51.268095 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.268060 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" event={"ID":"a464898b-d09d-4bb1-a02d-572875eca6ef","Type":"ContainerStarted","Data":"3c3dfe0e66125c8c9e25920454dfa8926d84dc7ba7af130c758a5408c91fcdb5"} Oct 30 08:03:51.268284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.268116 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" event={"ID":"a464898b-d09d-4bb1-a02d-572875eca6ef","Type":"ContainerStarted","Data":"04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013"} Oct 30 08:03:51.315091 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.315029 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" podStartSLOduration=2.315014031 podStartE2EDuration="2.315014031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:03:51.311728715 +0000 UTC m=+3995.002646151" watchObservedRunningTime="2026-10-30 08:03:51.315014031 +0000 UTC m=+3995.005931518" Oct 30 08:03:51.398202 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.398165 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q"] Oct 30 08:03:51.403550 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.403524 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.426391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.426358 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q"] Oct 30 08:03:51.494007 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.493970 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.494276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.494249 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzbwb\" (UniqueName: \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.494417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.494295 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.596326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.596283 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.596755 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.596439 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzbwb\" (UniqueName: \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.596755 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.596462 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.597194 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.597163 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.599103 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.599077 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.622601 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.622567 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzbwb\" (UniqueName: \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\") pod \"maas-api-revoke-sub-974b208a-bxh4q\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:51.783452 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:51.783414 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:52.104299 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:03:52.104235 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08226c3d_b049_4226_a27d_7f555fabc3ff.slice/crio-f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b WatchSource:0}: Error finding container f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b: Status 404 returned error can't find the container with id f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b Oct 30 08:03:52.104426 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:52.104316 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q"] Oct 30 08:03:52.278338 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:52.278301 5894 generic.go:356] "Generic (PLEG): container finished" podID="a464898b-d09d-4bb1-a02d-572875eca6ef" containerID="3c3dfe0e66125c8c9e25920454dfa8926d84dc7ba7af130c758a5408c91fcdb5" exitCode=0 Oct 30 08:03:52.278544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:52.278389 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" event={"ID":"a464898b-d09d-4bb1-a02d-572875eca6ef","Type":"ContainerDied","Data":"3c3dfe0e66125c8c9e25920454dfa8926d84dc7ba7af130c758a5408c91fcdb5"} Oct 30 08:03:52.280100 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:52.280068 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" event={"ID":"08226c3d-b049-4226-a27d-7f555fabc3ff","Type":"ContainerStarted","Data":"f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b"} Oct 30 08:03:52.949359 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:52.949322 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:53.291421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.291380 5894 generic.go:356] "Generic (PLEG): container finished" podID="08226c3d-b049-4226-a27d-7f555fabc3ff" containerID="962eaa56b5cb6bd4fcf6dd4452a16f03744abddb0e160b123a37424a9972dea0" exitCode=0 Oct 30 08:03:53.291621 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.291418 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" event={"ID":"08226c3d-b049-4226-a27d-7f555fabc3ff","Type":"ContainerDied","Data":"962eaa56b5cb6bd4fcf6dd4452a16f03744abddb0e160b123a37424a9972dea0"} Oct 30 08:03:53.458080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.458054 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:53.527167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.527074 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\" (UniqueName: \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\") pod \"a464898b-d09d-4bb1-a02d-572875eca6ef\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " Oct 30 08:03:53.527167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.527154 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\") pod \"a464898b-d09d-4bb1-a02d-572875eca6ef\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " Oct 30 08:03:53.527435 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.527418 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\") pod \"a464898b-d09d-4bb1-a02d-572875eca6ef\" (UID: \"a464898b-d09d-4bb1-a02d-572875eca6ef\") " Oct 30 08:03:53.527833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.527732 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca" pod "a464898b-d09d-4bb1-a02d-572875eca6ef" (UID: "a464898b-d09d-4bb1-a02d-572875eca6ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:53.528404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.528110 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:53.530027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.529999 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca" pod "a464898b-d09d-4bb1-a02d-572875eca6ef" (UID: "a464898b-d09d-4bb1-a02d-572875eca6ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:53.530144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.530103 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk" pod "a464898b-d09d-4bb1-a02d-572875eca6ef" (UID: "a464898b-d09d-4bb1-a02d-572875eca6ef"). InnerVolumeSpecName "kube-api-access-vf8qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:53.629619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.629582 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a464898b-d09d-4bb1-a02d-572875eca6ef-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:53.629619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:53.629614 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf8qk\" (UniqueName: \"kubernetes.io/projected/a464898b-d09d-4bb1-a02d-572875eca6ef-kube-api-access-vf8qk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.300575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.300541 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" Oct 30 08:03:54.300575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.300568 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6" event={"ID":"a464898b-d09d-4bb1-a02d-572875eca6ef","Type":"ContainerDied","Data":"04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013"} Oct 30 08:03:54.301124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.300617 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a04e7b247e317a9d5ff1cbb03a524f514b9afdcbded9bb6a0723645d32e013" Oct 30 08:03:54.474853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.474816 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:54.543360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.543306 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\") pod \"08226c3d-b049-4226-a27d-7f555fabc3ff\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " Oct 30 08:03:54.543582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.543384 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\") pod \"08226c3d-b049-4226-a27d-7f555fabc3ff\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " Oct 30 08:03:54.543582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.543563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\" (UniqueName: \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\") pod \"08226c3d-b049-4226-a27d-7f555fabc3ff\" (UID: \"08226c3d-b049-4226-a27d-7f555fabc3ff\") " Oct 30 08:03:54.544473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.544440 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca" pod "08226c3d-b049-4226-a27d-7f555fabc3ff" (UID: "08226c3d-b049-4226-a27d-7f555fabc3ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:03:54.546918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.546893 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca" pod "08226c3d-b049-4226-a27d-7f555fabc3ff" (UID: "08226c3d-b049-4226-a27d-7f555fabc3ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:03:54.547023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.546970 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb" pod "08226c3d-b049-4226-a27d-7f555fabc3ff" (UID: "08226c3d-b049-4226-a27d-7f555fabc3ff"). InnerVolumeSpecName "kube-api-access-qzbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:03:54.646048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.646016 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.646048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.646046 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08226c3d-b049-4226-a27d-7f555fabc3ff-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.646273 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.646059 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzbwb\" (UniqueName: \"kubernetes.io/projected/08226c3d-b049-4226-a27d-7f555fabc3ff-kube-api-access-qzbwb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:03:54.823996 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:54.823900 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:55.310444 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:55.310412 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" Oct 30 08:03:55.310902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:55.310447 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q" event={"ID":"08226c3d-b049-4226-a27d-7f555fabc3ff","Type":"ContainerDied","Data":"f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b"} Oct 30 08:03:55.310902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:55.310495 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b31b2ba2047b1100db9723757efe53a1332f2c3a92864afd43b2a415f6795b" Oct 30 08:03:56.436776 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:56.436742 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:57.445722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:57.445680 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:57.459160 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:57.459130 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:58.139108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:58.139074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:03:58.348646 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:58.348614 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:03:59.832426 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:03:59.832378 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:00.469055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:00.469017 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:00.583862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:00.583827 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:01.549923 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:01.549875 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:02.959408 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:02.959370 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:03.534383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:03.534335 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:03.663447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:03.663409 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:03.671545 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:03.671473 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj"] Oct 30 08:04:03.671777 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:03.671573 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-228571-j6vvj"] Oct 30 08:04:03.856535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:03.856441 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:04.263232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:04.263133 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:04.431974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:04.431936 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a175bd3-f38f-4407-ba4d-727c4f3918a6" path="/var/lib/kubelet/pods/6a175bd3-f38f-4407-ba4d-727c4f3918a6/volumes" Oct 30 08:04:04.829005 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:04:04.828966 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:04:05.617957 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:05.617921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:07.393896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:07.393859 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:09.236232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:09.236196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:10.959422 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:10.959385 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:11.172260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:11.172224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:11.434548 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:11.434509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:11.978386 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:11.978351 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:12.223277 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.223241 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9"] Oct 30 08:04:12.223960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.223938 5894 cpu_manager.go:414] "containerMap: removing container" podUID="08226c3d-b049-4226-a27d-7f555fabc3ff" containerName="revoke-keys" Oct 30 08:04:12.223960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.223960 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08226c3d-b049-4226-a27d-7f555fabc3ff" containerName="revoke-keys" Oct 30 08:04:12.224157 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.224020 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a464898b-d09d-4bb1-a02d-572875eca6ef" containerName="revoke-keys" Oct 30 08:04:12.224157 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.224030 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a464898b-d09d-4bb1-a02d-572875eca6ef" containerName="revoke-keys" Oct 30 08:04:12.224292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.224234 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="08226c3d-b049-4226-a27d-7f555fabc3ff" containerName="revoke-keys" Oct 30 08:04:12.224292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.224261 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a464898b-d09d-4bb1-a02d-572875eca6ef" containerName="revoke-keys" Oct 30 08:04:12.229344 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.229286 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.247672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.247629 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9"] Oct 30 08:04:12.258750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.258712 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:04:12.258750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.258715 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:04:12.258974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.258774 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:04:12.261328 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.261302 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:04:12.327182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.327145 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.327391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.327201 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.327391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.327291 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8c5d\" (UniqueName: \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.429229 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.429190 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.429229 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.429237 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.429493 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.429254 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8c5d\" (UniqueName: \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.429993 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.429970 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.431827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.431805 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.456369 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.456342 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8c5d\" (UniqueName: \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\") pod \"maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.543117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.543088 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:12.637223 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.637194 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:12.914140 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:12.914101 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9"] Oct 30 08:04:13.247016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:13.246970 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:13.480982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:13.480898 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" event={"ID":"53019d9c-9de2-4fb7-978a-5794ef0238d1","Type":"ContainerStarted","Data":"cbb117e94cc6c9bfa22ce29af09e2bbe3f8f6318596550d9dbe6e7bec218b668"} Oct 30 08:04:13.480982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:13.480953 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" event={"ID":"53019d9c-9de2-4fb7-978a-5794ef0238d1","Type":"ContainerStarted","Data":"fa48295006e72ecc34399435a397dd89cfebc43d100ca48dfa90d7b4673b5c93"} Oct 30 08:04:13.529160 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:13.529096 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" podStartSLOduration=1.529082236 podStartE2EDuration="1.529082236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:13.526383237 +0000 UTC m=+4017.217300677" watchObservedRunningTime="2026-10-30 08:04:13.529082236 +0000 UTC m=+4017.219999673" Oct 30 08:04:14.494374 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:14.494336 5894 generic.go:356] "Generic (PLEG): container finished" podID="53019d9c-9de2-4fb7-978a-5794ef0238d1" containerID="cbb117e94cc6c9bfa22ce29af09e2bbe3f8f6318596550d9dbe6e7bec218b668" exitCode=0 Oct 30 08:04:14.494897 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:14.494399 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" event={"ID":"53019d9c-9de2-4fb7-978a-5794ef0238d1","Type":"ContainerDied","Data":"cbb117e94cc6c9bfa22ce29af09e2bbe3f8f6318596550d9dbe6e7bec218b668"} Oct 30 08:04:15.149181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.149147 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:15.651669 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.651624 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:15.777778 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.777744 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\") pod \"53019d9c-9de2-4fb7-978a-5794ef0238d1\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " Oct 30 08:04:15.777994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.777847 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\" (UniqueName: \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\") pod \"53019d9c-9de2-4fb7-978a-5794ef0238d1\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " Oct 30 08:04:15.777994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.777910 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\") pod \"53019d9c-9de2-4fb7-978a-5794ef0238d1\" (UID: \"53019d9c-9de2-4fb7-978a-5794ef0238d1\") " Oct 30 08:04:15.778248 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.778220 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca" pod "53019d9c-9de2-4fb7-978a-5794ef0238d1" (UID: "53019d9c-9de2-4fb7-978a-5794ef0238d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:15.778517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.778495 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:15.780155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.780121 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca" pod "53019d9c-9de2-4fb7-978a-5794ef0238d1" (UID: "53019d9c-9de2-4fb7-978a-5794ef0238d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:15.780280 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.780155 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d" pod "53019d9c-9de2-4fb7-978a-5794ef0238d1" (UID: "53019d9c-9de2-4fb7-978a-5794ef0238d1"). InnerVolumeSpecName "kube-api-access-r8c5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:15.879509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.879471 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8c5d\" (UniqueName: \"kubernetes.io/projected/53019d9c-9de2-4fb7-978a-5794ef0238d1-kube-api-access-r8c5d\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:15.879509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:15.879501 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53019d9c-9de2-4fb7-978a-5794ef0238d1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:16.513234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:16.513205 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" Oct 30 08:04:16.513415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:16.513239 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9" event={"ID":"53019d9c-9de2-4fb7-978a-5794ef0238d1","Type":"ContainerDied","Data":"fa48295006e72ecc34399435a397dd89cfebc43d100ca48dfa90d7b4673b5c93"} Oct 30 08:04:16.513415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:16.513286 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa48295006e72ecc34399435a397dd89cfebc43d100ca48dfa90d7b4673b5c93" Oct 30 08:04:16.904866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:16.904827 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 08:04:17.033292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:17.033251 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:17.149718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:17.149674 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:19.055948 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:19.055913 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:20.981857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:20.981826 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:22.871549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:22.871511 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:22.993159 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:22.993115 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:24.718576 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:24.718541 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:26.249305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:26.249266 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:26.832510 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:26.832477 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:26.948477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:26.948436 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:26.977836 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:26.977798 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6"] Oct 30 08:04:26.991826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:26.991792 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-2761aa-nlsh6"] Oct 30 08:04:27.072690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:27.072653 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:27.545653 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:27.545612 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:28.434688 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:28.434653 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a464898b-d09d-4bb1-a02d-572875eca6ef" path="/var/lib/kubelet/pods/a464898b-d09d-4bb1-a02d-572875eca6ef/volumes" Oct 30 08:04:28.653105 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:28.653068 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:04:29.058801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:29.058764 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:31.028429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:31.028390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:31.928016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:31.927976 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" containerID="cri-o://c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca" gracePeriod=45 Oct 30 08:04:31.937827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:31.937789 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Oct 30 08:04:32.211751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.211720 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346077 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346188 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346373 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\" (UniqueName: \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346430 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346505 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.347544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.346544 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\") pod \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\" (UID: \"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00\") " Oct 30 08:04:32.348662 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.348529 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.349412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.349372 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.349699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.349662 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "kube-api-access-xxsbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:32.349937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.349906 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:32.350055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.349968 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.352391 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.352361 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm" pod "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" (UID: "0e8789b2-aaff-4d00-8c4b-66af9c2cfb00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:04:32.450196 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450159 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxsbj\" (UniqueName: \"kubernetes.io/projected/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-kube-api-access-xxsbj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.450196 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450198 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.450196 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450211 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.450449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450223 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.450449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450236 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.450449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.450247 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:32.684195 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.684161 5894 generic.go:356] "Generic (PLEG): container finished" podID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerID="c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca" exitCode=0 Oct 30 08:04:32.684404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.684202 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" event={"ID":"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00","Type":"ContainerDied","Data":"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca"} Oct 30 08:04:32.684404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.684208 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" Oct 30 08:04:32.684404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.684241 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56" event={"ID":"0e8789b2-aaff-4d00-8c4b-66af9c2cfb00","Type":"ContainerDied","Data":"9ecfc87092b75c5fa95f3f1c24b345aea8dfc2f6a8e3dead6d7f494d2473fcdd"} Oct 30 08:04:32.684404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.684264 5894 scope.go:122] "RemoveContainer" containerID="c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca" Oct 30 08:04:32.694745 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.694721 5894 scope.go:122] "RemoveContainer" containerID="c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca" Oct 30 08:04:32.695027 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:04:32.695007 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca\": container with ID starting with c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca not found: ID does not exist" containerID="c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca" Oct 30 08:04:32.695082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.695036 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca"} err="failed to get container status \"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca\": rpc error: code = NotFound desc = could not find container \"c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca\": container with ID starting with c203d3a38602121001e288cfba258e3e728481a0ce475e797c3217f032b742ca not found: ID does not exist" Oct 30 08:04:32.744404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.744298 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 08:04:32.758754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:32.758718 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b72d79/distinct-w0-b72d79-kserve-588f6f7948-tgb56"] Oct 30 08:04:33.208571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:33.208536 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:33.546951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:33.546907 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:34.432245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:34.432206 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" path="/var/lib/kubelet/pods/0e8789b2-aaff-4d00-8c4b-66af9c2cfb00/volumes" Oct 30 08:04:34.832704 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:04:34.832661 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:04:35.051432 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:35.051390 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:35.671021 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:35.670978 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:35.794506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:35.794455 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:35.996959 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:35.996850 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:36.643895 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.643863 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq"] Oct 30 08:04:36.644550 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644534 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" Oct 30 08:04:36.644601 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644553 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" Oct 30 08:04:36.644601 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644577 5894 cpu_manager.go:414] "containerMap: removing container" podUID="53019d9c-9de2-4fb7-978a-5794ef0238d1" containerName="revoke-keys" Oct 30 08:04:36.644601 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644585 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53019d9c-9de2-4fb7-978a-5794ef0238d1" containerName="revoke-keys" Oct 30 08:04:36.644779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644695 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8789b2-aaff-4d00-8c4b-66af9c2cfb00" containerName="main" Oct 30 08:04:36.644779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.644706 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="53019d9c-9de2-4fb7-978a-5794ef0238d1" containerName="revoke-keys" Oct 30 08:04:36.649974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.649950 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq"] Oct 30 08:04:36.650119 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.650076 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.654207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.654185 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.669020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.668994 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:04:36.672173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.672145 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:04:36.673880 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.673854 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq"] Oct 30 08:04:36.686806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.686769 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq"] Oct 30 08:04:36.689548 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.689520 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m"] Oct 30 08:04:36.693959 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.693935 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.712987 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.712954 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:04:36.714276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.714247 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:04:36.717868 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.717837 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m"] Oct 30 08:04:36.808782 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.808742 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j4mk\" (UniqueName: \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.808961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.808794 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.808961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.808873 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.809043 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.808966 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.809086 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.809058 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpcd\" (UniqueName: \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.809122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.809085 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.809180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.809166 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.809216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.809204 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.809261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.809239 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g97qq\" (UniqueName: \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.910471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910382 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.910471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910430 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.910471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910459 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.910863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910495 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g97qq\" (UniqueName: \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.910863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910655 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j4mk\" (UniqueName: \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.910863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910697 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.910863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910729 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.910863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910762 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.911168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.910897 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpcd\" (UniqueName: \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.911380 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.911354 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.911460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.911371 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.911715 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.911685 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.913308 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.913282 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.913489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.913467 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.913538 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.913467 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.936828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.936797 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g97qq\" (UniqueName: \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:36.937000 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.936975 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpcd\" (UniqueName: \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\") pod \"maas-api-revoke-sub-9c5213f7-6z29m\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:36.937050 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.937015 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j4mk\" (UniqueName: \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\") pod \"maas-api-revoke-sub-c08342fb-nq8gq\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.967769 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.967734 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:36.976342 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:36.976311 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:37.024060 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.024024 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:37.180905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.180805 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:37.293393 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:04:37.293356 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92721e74_7b49_4af6_9e98_91a8c2b4a374.slice/crio-f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394 WatchSource:0}: Error finding container f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394: Status 404 returned error can't find the container with id f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394 Oct 30 08:04:37.294161 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.294134 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq"] Oct 30 08:04:37.305595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.305566 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq"] Oct 30 08:04:37.307287 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:04:37.307257 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf896c529_88ca_4783_b497_dad572c71ddf.slice/crio-22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44 WatchSource:0}: Error finding container 22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44: Status 404 returned error can't find the container with id 22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44 Oct 30 08:04:37.365690 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:04:37.365620 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32e3508c_824e_4326_bb13_8d3b598d5ff9.slice/crio-52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a WatchSource:0}: Error finding container 52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a: Status 404 returned error can't find the container with id 52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a Oct 30 08:04:37.365918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.365895 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m"] Oct 30 08:04:37.734422 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.734317 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" event={"ID":"f896c529-88ca-4783-b497-dad572c71ddf","Type":"ContainerStarted","Data":"f5bcd3c32d8f0b034d45829a467001ac1b89b2f7b6b8893eb015489fb23408c1"} Oct 30 08:04:37.734422 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.734378 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" event={"ID":"f896c529-88ca-4783-b497-dad572c71ddf","Type":"ContainerStarted","Data":"22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44"} Oct 30 08:04:37.735757 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.735727 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" event={"ID":"92721e74-7b49-4af6-9e98-91a8c2b4a374","Type":"ContainerStarted","Data":"f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394"} Oct 30 08:04:37.737136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.737114 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" event={"ID":"32e3508c-824e-4326-bb13-8d3b598d5ff9","Type":"ContainerStarted","Data":"52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a"} Oct 30 08:04:37.780574 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:37.780530 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" podStartSLOduration=1.780514611 podStartE2EDuration="1.780514611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:04:37.779101398 +0000 UTC m=+4041.470018836" watchObservedRunningTime="2026-10-30 08:04:37.780514611 +0000 UTC m=+4041.471432066" Oct 30 08:04:38.750237 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.750146 5894 generic.go:356] "Generic (PLEG): container finished" podID="f896c529-88ca-4783-b497-dad572c71ddf" containerID="f5bcd3c32d8f0b034d45829a467001ac1b89b2f7b6b8893eb015489fb23408c1" exitCode=0 Oct 30 08:04:38.750754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.750238 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" event={"ID":"f896c529-88ca-4783-b497-dad572c71ddf","Type":"ContainerDied","Data":"f5bcd3c32d8f0b034d45829a467001ac1b89b2f7b6b8893eb015489fb23408c1"} Oct 30 08:04:38.752107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.752083 5894 generic.go:356] "Generic (PLEG): container finished" podID="92721e74-7b49-4af6-9e98-91a8c2b4a374" containerID="c08a2dcf5872372367c8de6fdc9486b3821cd64baa523c850915afea0dd24e84" exitCode=0 Oct 30 08:04:38.752250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.752110 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" event={"ID":"92721e74-7b49-4af6-9e98-91a8c2b4a374","Type":"ContainerDied","Data":"c08a2dcf5872372367c8de6fdc9486b3821cd64baa523c850915afea0dd24e84"} Oct 30 08:04:38.753957 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.753932 5894 generic.go:356] "Generic (PLEG): container finished" podID="32e3508c-824e-4326-bb13-8d3b598d5ff9" containerID="8d4a0804e370ea7034019bbd52d51784d1397a71d4398fd889195f487146ee38" exitCode=0 Oct 30 08:04:38.754051 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:38.754007 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" event={"ID":"32e3508c-824e-4326-bb13-8d3b598d5ff9","Type":"ContainerDied","Data":"8d4a0804e370ea7034019bbd52d51784d1397a71d4398fd889195f487146ee38"} Oct 30 08:04:39.061435 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.061395 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:39.937027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.936248 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:39.966936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.966905 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\") pod \"32e3508c-824e-4326-bb13-8d3b598d5ff9\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " Oct 30 08:04:39.967101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.967011 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\") pod \"32e3508c-824e-4326-bb13-8d3b598d5ff9\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " Oct 30 08:04:39.967320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.967296 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca" pod "32e3508c-824e-4326-bb13-8d3b598d5ff9" (UID: "32e3508c-824e-4326-bb13-8d3b598d5ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:39.967527 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.967505 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:39.969367 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.969336 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca" pod "32e3508c-824e-4326-bb13-8d3b598d5ff9" (UID: "32e3508c-824e-4326-bb13-8d3b598d5ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:39.973599 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.973576 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:39.990874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:39.990846 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:40.068492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068448 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\") pod \"92721e74-7b49-4af6-9e98-91a8c2b4a374\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " Oct 30 08:04:40.068721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068502 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\" (UniqueName: \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\") pod \"f896c529-88ca-4783-b497-dad572c71ddf\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " Oct 30 08:04:40.068721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068616 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\" (UniqueName: \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\") pod \"32e3508c-824e-4326-bb13-8d3b598d5ff9\" (UID: \"32e3508c-824e-4326-bb13-8d3b598d5ff9\") " Oct 30 08:04:40.068721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068699 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\" (UniqueName: \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\") pod \"92721e74-7b49-4af6-9e98-91a8c2b4a374\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " Oct 30 08:04:40.069003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068738 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\") pod \"92721e74-7b49-4af6-9e98-91a8c2b4a374\" (UID: \"92721e74-7b49-4af6-9e98-91a8c2b4a374\") " Oct 30 08:04:40.069003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068788 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\") pod \"f896c529-88ca-4783-b497-dad572c71ddf\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " Oct 30 08:04:40.069003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.068822 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\") pod \"f896c529-88ca-4783-b497-dad572c71ddf\" (UID: \"f896c529-88ca-4783-b497-dad572c71ddf\") " Oct 30 08:04:40.069618 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.069350 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca" pod "f896c529-88ca-4783-b497-dad572c71ddf" (UID: "f896c529-88ca-4783-b497-dad572c71ddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:40.069618 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.069545 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32e3508c-824e-4326-bb13-8d3b598d5ff9-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.069618 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.069568 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.070259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.070213 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca" pod "92721e74-7b49-4af6-9e98-91a8c2b4a374" (UID: "92721e74-7b49-4af6-9e98-91a8c2b4a374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:04:40.071423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.071399 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk" pod "92721e74-7b49-4af6-9e98-91a8c2b4a374" (UID: "92721e74-7b49-4af6-9e98-91a8c2b4a374"). InnerVolumeSpecName "kube-api-access-7j4mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:40.071556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.071523 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca" pod "f896c529-88ca-4783-b497-dad572c71ddf" (UID: "f896c529-88ca-4783-b497-dad572c71ddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:40.071752 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.071729 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd" pod "32e3508c-824e-4326-bb13-8d3b598d5ff9" (UID: "32e3508c-824e-4326-bb13-8d3b598d5ff9"). InnerVolumeSpecName "kube-api-access-qhpcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:40.072471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.072452 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca" pod "92721e74-7b49-4af6-9e98-91a8c2b4a374" (UID: "92721e74-7b49-4af6-9e98-91a8c2b4a374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:04:40.074513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.074480 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq" pod "f896c529-88ca-4783-b497-dad572c71ddf" (UID: "f896c529-88ca-4783-b497-dad572c71ddf"). InnerVolumeSpecName "kube-api-access-g97qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:04:40.171893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171858 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpcd\" (UniqueName: \"kubernetes.io/projected/32e3508c-824e-4326-bb13-8d3b598d5ff9-kube-api-access-qhpcd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.171893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171889 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j4mk\" (UniqueName: \"kubernetes.io/projected/92721e74-7b49-4af6-9e98-91a8c2b4a374-kube-api-access-7j4mk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.172116 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171920 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.172116 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171931 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f896c529-88ca-4783-b497-dad572c71ddf-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.172116 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171941 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92721e74-7b49-4af6-9e98-91a8c2b4a374-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.172116 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.171949 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g97qq\" (UniqueName: \"kubernetes.io/projected/f896c529-88ca-4783-b497-dad572c71ddf-kube-api-access-g97qq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:04:40.772604 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.772512 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" Oct 30 08:04:40.772821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.772517 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq" event={"ID":"f896c529-88ca-4783-b497-dad572c71ddf","Type":"ContainerDied","Data":"22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44"} Oct 30 08:04:40.772821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.772688 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22c7190009dfaf934122a0d0ecc7df31a285cfeb4c47ef15ca0284c92f1b4c44" Oct 30 08:04:40.774169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.774139 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" Oct 30 08:04:40.774299 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.774198 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq" event={"ID":"92721e74-7b49-4af6-9e98-91a8c2b4a374","Type":"ContainerDied","Data":"f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394"} Oct 30 08:04:40.774299 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.774246 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8d4ee61fccc629c36667df4c9e81a24cddf969bca319b19a8cc132aa8e9a394" Oct 30 08:04:40.775884 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.775859 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" event={"ID":"32e3508c-824e-4326-bb13-8d3b598d5ff9","Type":"ContainerDied","Data":"52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a"} Oct 30 08:04:40.776015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.775903 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m" Oct 30 08:04:40.776015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:40.775905 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b270f2f5ad2e72e758012e7a9210f9952e2a09387b018a0a533c752455f78a" Oct 30 08:04:41.019306 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:41.018570 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:42.876775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:42.876741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:44.637890 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:44.637849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:46.399394 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:46.399355 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:47.295360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:47.295319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:47.422082 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:47.422042 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:48.388606 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:48.388558 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:50.332236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:50.332192 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:52.335578 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:52.335531 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:54.286260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:54.286216 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:54.410864 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:54.410587 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:54.533979 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:54.533944 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:54.750571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:54.750493 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:54.949543 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:54.949509 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:56.147161 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:56.147125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:56.262291 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:56.262245 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:57.903972 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:57.903883 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:04:59.799652 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:04:59.799550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:01.597538 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:01.597501 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:02.549954 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:02.549905 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:02.668879 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:02.668849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:03.556998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:03.556958 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:04.828414 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:05:04.828374 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:05:05.111003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:05.110906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:05.556267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:05.556222 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:07.183825 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:07.183786 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:07.729304 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:07.729247 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:07.848853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:07.848814 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:07.876074 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:07.876037 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq"] Oct 30 08:05:07.922057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:07.922017 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-4b0e81-72jvq"] Oct 30 08:05:08.372238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:08.372200 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:08.434406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:08.434362 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f896c529-88ca-4783-b497-dad572c71ddf" path="/var/lib/kubelet/pods/f896c529-88ca-4783-b497-dad572c71ddf/volumes" Oct 30 08:05:09.612680 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:09.612609 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:10.231492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.231457 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:10.366239 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.366198 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:10.377251 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.377193 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9"] Oct 30 08:05:10.382487 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.382453 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b72d79-qglc9"] Oct 30 08:05:10.431522 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.431484 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53019d9c-9de2-4fb7-978a-5794ef0238d1" path="/var/lib/kubelet/pods/53019d9c-9de2-4fb7-978a-5794ef0238d1/volumes" Oct 30 08:05:10.497828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.497515 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:10.937193 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:10.937152 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:12.435166 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:12.435132 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:14.316920 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:14.316881 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:14.939210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:14.939176 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:15.060099 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:15.060063 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:15.278733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:15.278690 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:15.777434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:15.777406 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:16.044931 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.044843 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x"] Oct 30 08:05:16.045564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045541 5894 cpu_manager.go:414] "containerMap: removing container" podUID="32e3508c-824e-4326-bb13-8d3b598d5ff9" containerName="revoke-keys" Oct 30 08:05:16.045730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045567 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32e3508c-824e-4326-bb13-8d3b598d5ff9" containerName="revoke-keys" Oct 30 08:05:16.045730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045615 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f896c529-88ca-4783-b497-dad572c71ddf" containerName="revoke-keys" Oct 30 08:05:16.045730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045624 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f896c529-88ca-4783-b497-dad572c71ddf" containerName="revoke-keys" Oct 30 08:05:16.045730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045666 5894 cpu_manager.go:414] "containerMap: removing container" podUID="92721e74-7b49-4af6-9e98-91a8c2b4a374" containerName="revoke-keys" Oct 30 08:05:16.045730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045674 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92721e74-7b49-4af6-9e98-91a8c2b4a374" containerName="revoke-keys" Oct 30 08:05:16.046028 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045846 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="92721e74-7b49-4af6-9e98-91a8c2b4a374" containerName="revoke-keys" Oct 30 08:05:16.046028 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045870 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="32e3508c-824e-4326-bb13-8d3b598d5ff9" containerName="revoke-keys" Oct 30 08:05:16.046028 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.045890 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f896c529-88ca-4783-b497-dad572c71ddf" containerName="revoke-keys" Oct 30 08:05:16.051101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.051075 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.069230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.069202 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x"] Oct 30 08:05:16.080992 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.080961 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:05:16.081172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.081058 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:05:16.124921 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.124889 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:05:16.124921 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.124898 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:05:16.222128 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.222089 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.222378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.222171 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.222490 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.222396 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8s8\" (UniqueName: \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.324084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.323990 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.324287 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.324146 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8s8\" (UniqueName: \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.324363 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.324294 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.325204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.325163 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.326730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.326702 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.350223 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.350194 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8s8\" (UniqueName: \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.365720 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.365692 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:16.460312 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.460195 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:16.674491 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:05:16.674444 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7d0301_446a_44e6_b234_a77e099bdeb5.slice/crio-e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7 WatchSource:0}: Error finding container e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7: Status 404 returned error can't find the container with id e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7 Oct 30 08:05:16.675519 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:16.674990 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x"] Oct 30 08:05:17.079895 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:17.079859 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" event={"ID":"ce7d0301-446a-44e6-b234-a77e099bdeb5","Type":"ContainerStarted","Data":"620af1c9607f5840018801f544218c7c9029e0cad4fc368922eb93accfa38160"} Oct 30 08:05:17.080371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:17.079912 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" event={"ID":"ce7d0301-446a-44e6-b234-a77e099bdeb5","Type":"ContainerStarted","Data":"e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7"} Oct 30 08:05:17.126383 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:17.126321 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" podStartSLOduration=1.126306214 podStartE2EDuration="1.126306214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:17.124982508 +0000 UTC m=+4080.815899947" watchObservedRunningTime="2026-10-30 08:05:17.126306214 +0000 UTC m=+4080.817223641" Oct 30 08:05:18.092678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:18.092617 5894 generic.go:356] "Generic (PLEG): container finished" podID="ce7d0301-446a-44e6-b234-a77e099bdeb5" containerID="620af1c9607f5840018801f544218c7c9029e0cad4fc368922eb93accfa38160" exitCode=0 Oct 30 08:05:18.092678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:18.092673 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" event={"ID":"ce7d0301-446a-44e6-b234-a77e099bdeb5","Type":"ContainerDied","Data":"620af1c9607f5840018801f544218c7c9029e0cad4fc368922eb93accfa38160"} Oct 30 08:05:18.597289 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:18.597242 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:18.715926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:18.715887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:19.042809 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.042774 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:19.316607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.316580 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:19.478186 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.478138 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\") pod \"ce7d0301-446a-44e6-b234-a77e099bdeb5\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " Oct 30 08:05:19.478384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.478218 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\" (UniqueName: \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\") pod \"ce7d0301-446a-44e6-b234-a77e099bdeb5\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " Oct 30 08:05:19.478384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.478264 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\") pod \"ce7d0301-446a-44e6-b234-a77e099bdeb5\" (UID: \"ce7d0301-446a-44e6-b234-a77e099bdeb5\") " Oct 30 08:05:19.478592 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.478561 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca" pod "ce7d0301-446a-44e6-b234-a77e099bdeb5" (UID: "ce7d0301-446a-44e6-b234-a77e099bdeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:19.478802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.478782 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:19.480456 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.480431 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca" pod "ce7d0301-446a-44e6-b234-a77e099bdeb5" (UID: "ce7d0301-446a-44e6-b234-a77e099bdeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:19.480552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.480531 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8" pod "ce7d0301-446a-44e6-b234-a77e099bdeb5" (UID: "ce7d0301-446a-44e6-b234-a77e099bdeb5"). InnerVolumeSpecName "kube-api-access-sx8s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:19.580376 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.580289 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx8s8\" (UniqueName: \"kubernetes.io/projected/ce7d0301-446a-44e6-b234-a77e099bdeb5-kube-api-access-sx8s8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:19.580543 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:19.580393 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce7d0301-446a-44e6-b234-a77e099bdeb5-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:20.109330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:20.109291 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" Oct 30 08:05:20.109535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:20.109353 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x" event={"ID":"ce7d0301-446a-44e6-b234-a77e099bdeb5","Type":"ContainerDied","Data":"e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7"} Oct 30 08:05:20.109535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:20.109426 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e33cde4bd5e9ea42f3114e97ecf8e9203b40a9a7ffb6e81b0ac9b16a2e665ac7" Oct 30 08:05:20.432320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:20.432225 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:20.837267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:20.837230 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:21.723225 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.723190 5894 scope.go:122] "RemoveContainer" containerID="b069e44b2908871d7b2c74bf20e5d96e82a37e4be244831d4158e95dbab9ccc2" Oct 30 08:05:21.734299 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.734272 5894 scope.go:122] "RemoveContainer" containerID="2eb731023d2b7fb5de7484c1d254e56b992814a6b8632f511d5b41cadc389695" Oct 30 08:05:21.744840 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.744811 5894 scope.go:122] "RemoveContainer" containerID="acb5a6ae6c211cf6e4b3154c5e0ed931ca6e9dea424cec3b553498d7309326e6" Oct 30 08:05:21.757127 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.757103 5894 scope.go:122] "RemoveContainer" containerID="d3126acceb77794669ac0ec36fc5d7d607494d2d6106d98d77411711b77b1c7f" Oct 30 08:05:21.768995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.768969 5894 scope.go:122] "RemoveContainer" containerID="17f4c68b8a55e63a9cd77819d2fff85866ff301768c01e4fda7be324a0dfe100" Oct 30 08:05:21.779125 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.779103 5894 scope.go:122] "RemoveContainer" containerID="c1f9bd115ea990fd0b68a8329fdc935fcfa1d118690ccbbd9b6ae9e4524240e3" Oct 30 08:05:21.788612 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:21.788589 5894 scope.go:122] "RemoveContainer" containerID="f55c7ca23390bba35c5fb18db607eaeff34f0e0d03444b316712665020982673" Oct 30 08:05:22.346974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:22.346933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:23.044008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:23.043973 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:05:24.033341 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:24.033296 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx"] Oct 30 08:05:24.047917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:24.047884 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa23cd32-s8brx"] Oct 30 08:05:24.116103 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:24.116067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:24.432507 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:24.432415 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e445752-1cf8-43cc-b495-61da204d5ee6" path="/var/lib/kubelet/pods/9e445752-1cf8-43cc-b495-61da204d5ee6/volumes" Oct 30 08:05:25.975097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:25.975061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:26.847084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:26.847048 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:26.962258 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:26.962219 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:27.863130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:27.863095 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:29.678135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:29.678099 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:31.460047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:31.460010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:33.342855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:33.342815 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:33.662066 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:33.661971 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:34.829498 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:05:34.829456 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:05:35.136158 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:35.136043 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:37.023791 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:37.023755 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:38.785611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:38.785563 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:40.602975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:40.602933 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:42.442164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:42.442126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:44.256371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:44.256336 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:46.144926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:46.144887 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:47.918960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:47.918919 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:49.826152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:49.826106 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:51.711152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:51.711114 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:53.643371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:53.643329 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:55.118001 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.117970 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v"] Oct 30 08:05:55.118670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.118616 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ce7d0301-446a-44e6-b234-a77e099bdeb5" containerName="revoke-keys" Oct 30 08:05:55.118670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.118652 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce7d0301-446a-44e6-b234-a77e099bdeb5" containerName="revoke-keys" Oct 30 08:05:55.118794 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.118784 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce7d0301-446a-44e6-b234-a77e099bdeb5" containerName="revoke-keys" Oct 30 08:05:55.122397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.122374 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.143549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.143517 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl"] Oct 30 08:05:55.147802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.147777 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.152955 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.152927 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:05:55.155850 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.155821 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:05:55.156314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.155921 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:05:55.156314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.155986 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:05:55.176767 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.176728 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v"] Oct 30 08:05:55.179156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.179127 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl"] Oct 30 08:05:55.273456 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273419 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.273650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273483 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.273650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273595 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.273750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273726 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpd8\" (UniqueName: \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.273826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273807 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x5n\" (UniqueName: \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.273865 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.273842 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.375232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375145 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x5n\" (UniqueName: \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.375232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375193 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.375508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375250 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.375508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375297 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.375508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375347 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.375508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.375476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpd8\" (UniqueName: \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.376138 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.376109 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.376277 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.376258 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.377925 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.377898 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.378033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.378018 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.401563 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.401531 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpd8\" (UniqueName: \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\") pod \"maas-api-revoke-sub-4c66bf60-v288v\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.401733 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.401620 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x5n\" (UniqueName: \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\") pod \"maas-api-revoke-sub-e67fd566-t46nl\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.437815 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.437784 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:55.469423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.469389 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:55.488534 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.488500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:55.750866 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:05:55.750836 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3050a307_8385_4ecf_b84a_20f6396d8afb.slice/crio-e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15 WatchSource:0}: Error finding container e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15: Status 404 returned error can't find the container with id e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15 Oct 30 08:05:55.751722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.751698 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v"] Oct 30 08:05:55.822161 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:05:55.822126 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b9a27a_647a_4098_94ca_7372355d9282.slice/crio-630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b WatchSource:0}: Error finding container 630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b: Status 404 returned error can't find the container with id 630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b Oct 30 08:05:55.822622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:55.822598 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl"] Oct 30 08:05:56.378226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.378140 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" event={"ID":"53b9a27a-647a-4098-94ca-7372355d9282","Type":"ContainerStarted","Data":"77a9ea8ad32a01e75ca0015bbabd806af971a5a27b0967b731a432618c1e8b1c"} Oct 30 08:05:56.378226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.378204 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" event={"ID":"53b9a27a-647a-4098-94ca-7372355d9282","Type":"ContainerStarted","Data":"630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b"} Oct 30 08:05:56.380267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.380237 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" event={"ID":"3050a307-8385-4ecf-b84a-20f6396d8afb","Type":"ContainerStarted","Data":"ab0b0ad9d811142956fbbf9fa7a0c66ae520deffc7d0357d8223e4c550300d33"} Oct 30 08:05:56.380393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.380300 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" event={"ID":"3050a307-8385-4ecf-b84a-20f6396d8afb","Type":"ContainerStarted","Data":"e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15"} Oct 30 08:05:56.425118 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.425053 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" podStartSLOduration=1.425038692 podStartE2EDuration="1.425038692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:56.423514836 +0000 UTC m=+4120.114432264" watchObservedRunningTime="2026-10-30 08:05:56.425038692 +0000 UTC m=+4120.115956128" Oct 30 08:05:56.480650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:56.480578 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" podStartSLOduration=1.480561658 podStartE2EDuration="1.480561658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:05:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:05:56.477800791 +0000 UTC m=+4120.168718230" watchObservedRunningTime="2026-10-30 08:05:56.480561658 +0000 UTC m=+4120.171479086" Oct 30 08:05:57.345746 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:57.345705 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:57.399737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:57.399695 5894 generic.go:356] "Generic (PLEG): container finished" podID="3050a307-8385-4ecf-b84a-20f6396d8afb" containerID="ab0b0ad9d811142956fbbf9fa7a0c66ae520deffc7d0357d8223e4c550300d33" exitCode=0 Oct 30 08:05:57.400267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:57.399897 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" event={"ID":"3050a307-8385-4ecf-b84a-20f6396d8afb","Type":"ContainerDied","Data":"ab0b0ad9d811142956fbbf9fa7a0c66ae520deffc7d0357d8223e4c550300d33"} Oct 30 08:05:57.403040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:57.403004 5894 generic.go:356] "Generic (PLEG): container finished" podID="53b9a27a-647a-4098-94ca-7372355d9282" containerID="77a9ea8ad32a01e75ca0015bbabd806af971a5a27b0967b731a432618c1e8b1c" exitCode=0 Oct 30 08:05:57.403214 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:57.403088 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" event={"ID":"53b9a27a-647a-4098-94ca-7372355d9282","Type":"ContainerDied","Data":"77a9ea8ad32a01e75ca0015bbabd806af971a5a27b0967b731a432618c1e8b1c"} Oct 30 08:05:58.581378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.581352 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:58.621126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.621098 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\") pod \"3050a307-8385-4ecf-b84a-20f6396d8afb\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " Oct 30 08:05:58.621286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.621138 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\") pod \"3050a307-8385-4ecf-b84a-20f6396d8afb\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " Oct 30 08:05:58.621286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.621230 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\" (UniqueName: \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\") pod \"3050a307-8385-4ecf-b84a-20f6396d8afb\" (UID: \"3050a307-8385-4ecf-b84a-20f6396d8afb\") " Oct 30 08:05:58.622001 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.621969 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca" pod "3050a307-8385-4ecf-b84a-20f6396d8afb" (UID: "3050a307-8385-4ecf-b84a-20f6396d8afb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:58.623947 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.623922 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8" pod "3050a307-8385-4ecf-b84a-20f6396d8afb" (UID: "3050a307-8385-4ecf-b84a-20f6396d8afb"). InnerVolumeSpecName "kube-api-access-8rpd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:58.624686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.624664 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca" pod "3050a307-8385-4ecf-b84a-20f6396d8afb" (UID: "3050a307-8385-4ecf-b84a-20f6396d8afb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:58.651539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.651503 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.722898 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\") pod \"53b9a27a-647a-4098-94ca-7372355d9282\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723015 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\") pod \"53b9a27a-647a-4098-94ca-7372355d9282\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723087 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\" (UniqueName: \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\") pod \"53b9a27a-647a-4098-94ca-7372355d9282\" (UID: \"53b9a27a-647a-4098-94ca-7372355d9282\") " Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723324 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca" pod "53b9a27a-647a-4098-94ca-7372355d9282" (UID: "53b9a27a-647a-4098-94ca-7372355d9282"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723683 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723706 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3050a307-8385-4ecf-b84a-20f6396d8afb-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723721 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpd8\" (UniqueName: \"kubernetes.io/projected/3050a307-8385-4ecf-b84a-20f6396d8afb-kube-api-access-8rpd8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.723735 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.725903 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n" pod "53b9a27a-647a-4098-94ca-7372355d9282" (UID: "53b9a27a-647a-4098-94ca-7372355d9282"). InnerVolumeSpecName "kube-api-access-b8x5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:05:58.738750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.725983 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca" pod "53b9a27a-647a-4098-94ca-7372355d9282" (UID: "53b9a27a-647a-4098-94ca-7372355d9282"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:05:58.743070 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.743036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:58.835691 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.834723 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8x5n\" (UniqueName: \"kubernetes.io/projected/53b9a27a-647a-4098-94ca-7372355d9282-kube-api-access-b8x5n\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:58.835691 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:58.834763 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53b9a27a-647a-4098-94ca-7372355d9282-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:05:59.098663 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.097086 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:59.370057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.369952 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:59.433264 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.433205 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" event={"ID":"3050a307-8385-4ecf-b84a-20f6396d8afb","Type":"ContainerDied","Data":"e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15"} Oct 30 08:05:59.433614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.433290 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e59172a9bf6978d6292ec5a33d7463cc616c4dd2d25aa65572467e92d297ad15" Oct 30 08:05:59.433614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.433401 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v" Oct 30 08:05:59.446596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.446563 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" Oct 30 08:05:59.446814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.446590 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl" event={"ID":"53b9a27a-647a-4098-94ca-7372355d9282","Type":"ContainerDied","Data":"630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b"} Oct 30 08:05:59.446814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.446693 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="630394dae02d7a2de77cdc24142d8bda10241e0c9b9490652ca5676e82debf4b" Oct 30 08:05:59.501706 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.501668 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:05:59.634089 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:05:59.633981 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:00.736096 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:00.736055 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:01.303305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:01.303267 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:03.146882 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:03.146836 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:04.828867 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:06:04.828831 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:06:04.953261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:04.953226 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:05.610535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:05.610487 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:06:06.277258 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:06.277216 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 08:06:06.277781 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:06.277470 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" containerID="cri-o://d34b6a0baed90b366cd5f93bb8d8f0d830e18da4535875e9be911038f3b26fd2" gracePeriod=30 Oct 30 08:06:06.871735 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:06.871697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:08.756761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:08.756723 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:09.677555 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.677522 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:09.807574 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.807531 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:09.904108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904077 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq"] Oct 30 08:06:09.904751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904736 5894 cpu_manager.go:414] "containerMap: removing container" podUID="53b9a27a-647a-4098-94ca-7372355d9282" containerName="revoke-keys" Oct 30 08:06:09.904806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904752 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b9a27a-647a-4098-94ca-7372355d9282" containerName="revoke-keys" Oct 30 08:06:09.904806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904772 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3050a307-8385-4ecf-b84a-20f6396d8afb" containerName="revoke-keys" Oct 30 08:06:09.904806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904777 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3050a307-8385-4ecf-b84a-20f6396d8afb" containerName="revoke-keys" Oct 30 08:06:09.904917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904907 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b9a27a-647a-4098-94ca-7372355d9282" containerName="revoke-keys" Oct 30 08:06:09.904963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.904923 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3050a307-8385-4ecf-b84a-20f6396d8afb" containerName="revoke-keys" Oct 30 08:06:09.908711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.908690 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:09.928363 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.928291 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:06:09.931974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.931941 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq"] Oct 30 08:06:09.941927 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.941880 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:06:09.942950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.942921 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:06:09.943112 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.942955 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:06:09.994700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.994665 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:09.994890 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.994800 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjcpd\" (UniqueName: \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:09.994936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:09.994892 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.095897 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.095856 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjcpd\" (UniqueName: \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.096113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.095969 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.096184 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.096172 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.096736 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.096714 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.098585 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.098566 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.122169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.122136 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjcpd\" (UniqueName: \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\") pod \"maas-api-revoke-sub-318d39cf-bznnq\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.223629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.223548 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:10.530276 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:10.530244 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode21f34cf_b472_4fad_a8c4_b2ee6af46079.slice/crio-f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366 WatchSource:0}: Error finding container f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366: Status 404 returned error can't find the container with id f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366 Oct 30 08:06:10.530402 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.530272 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq"] Oct 30 08:06:10.657236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:10.657201 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:11.544346 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.544307 5894 generic.go:356] "Generic (PLEG): container finished" podID="e21f34cf-b472-4fad-a8c4-b2ee6af46079" containerID="aa32ab67097671801cd6584bdcef7e10d4ee62f1aa6df7fb37b8b685bc9523d5" exitCode=0 Oct 30 08:06:11.544854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.544401 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" event={"ID":"e21f34cf-b472-4fad-a8c4-b2ee6af46079","Type":"ContainerDied","Data":"aa32ab67097671801cd6584bdcef7e10d4ee62f1aa6df7fb37b8b685bc9523d5"} Oct 30 08:06:11.544854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.544510 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" event={"ID":"e21f34cf-b472-4fad-a8c4-b2ee6af46079","Type":"ContainerStarted","Data":"f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366"} Oct 30 08:06:11.546077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.546055 5894 generic.go:356] "Generic (PLEG): container finished" podID="997beedc-265c-4542-b38e-100bdae82e19" containerID="d34b6a0baed90b366cd5f93bb8d8f0d830e18da4535875e9be911038f3b26fd2" exitCode=0 Oct 30 08:06:11.546207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.546131 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" event={"ID":"997beedc-265c-4542-b38e-100bdae82e19","Type":"ContainerDied","Data":"d34b6a0baed90b366cd5f93bb8d8f0d830e18da4535875e9be911038f3b26fd2"} Oct 30 08:06:11.546207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.546182 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" event={"ID":"997beedc-265c-4542-b38e-100bdae82e19","Type":"ContainerDied","Data":"b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa"} Oct 30 08:06:11.546207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.546199 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80fb28cb44eb6e1c1a5dfc0dbb7dbee0153ea23c05cb2d57c852e04bec60faa" Oct 30 08:06:11.562764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.562690 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 08:06:11.723134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723097 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723333 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723176 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723333 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723207 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723333 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723267 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723532 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723339 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723532 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723462 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723532 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723490 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723541 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.723749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723547 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.723749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723605 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:11.723749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723647 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.723973 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723939 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.724043 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.723991 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\") pod \"997beedc-265c-4542-b38e-100bdae82e19\" (UID: \"997beedc-265c-4542-b38e-100bdae82e19\") " Oct 30 08:06:11.724103 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.724063 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.725003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.724967 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/997beedc-265c-4542-b38e-100bdae82e19-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.725003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.724999 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.725210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.725013 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.725210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.725027 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.725210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.725040 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.726568 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.726532 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "kube-api-access-bx54q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.726734 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.726587 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:11.726734 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.726723 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:11.726878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.726781 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token" pod "997beedc-265c-4542-b38e-100bdae82e19" (UID: "997beedc-265c-4542-b38e-100bdae82e19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:11.826286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.826201 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/997beedc-265c-4542-b38e-100bdae82e19-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.826286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.826231 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/997beedc-265c-4542-b38e-100bdae82e19-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.826286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.826240 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx54q\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-kube-api-access-bx54q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:11.826286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:11.826251 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/997beedc-265c-4542-b38e-100bdae82e19-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:12.555059 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.555026 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl" Oct 30 08:06:12.575693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.575652 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:12.638751 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.638704 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 08:06:12.642106 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.642076 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-v7mpl"] Oct 30 08:06:12.718835 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.718802 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:12.840074 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.840033 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\") pod \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " Oct 30 08:06:12.840259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.840240 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\" (UniqueName: \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\") pod \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " Oct 30 08:06:12.840360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.840346 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\") pod \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\" (UID: \"e21f34cf-b472-4fad-a8c4-b2ee6af46079\") " Oct 30 08:06:12.840446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.840417 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca" pod "e21f34cf-b472-4fad-a8c4-b2ee6af46079" (UID: "e21f34cf-b472-4fad-a8c4-b2ee6af46079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:12.840763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.840747 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:12.842403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.842361 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca" pod "e21f34cf-b472-4fad-a8c4-b2ee6af46079" (UID: "e21f34cf-b472-4fad-a8c4-b2ee6af46079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:12.842543 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.842406 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd" pod "e21f34cf-b472-4fad-a8c4-b2ee6af46079" (UID: "e21f34cf-b472-4fad-a8c4-b2ee6af46079"). InnerVolumeSpecName "kube-api-access-mjcpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:12.941961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.941924 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjcpd\" (UniqueName: \"kubernetes.io/projected/e21f34cf-b472-4fad-a8c4-b2ee6af46079-kube-api-access-mjcpd\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:12.941961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:12.941958 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e21f34cf-b472-4fad-a8c4-b2ee6af46079-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:13.564594 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:13.564560 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" Oct 30 08:06:13.565115 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:13.564595 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318d39cf-bznnq" event={"ID":"e21f34cf-b472-4fad-a8c4-b2ee6af46079","Type":"ContainerDied","Data":"f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366"} Oct 30 08:06:13.565115 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:13.564678 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f57d79308caa394d866d0297580741f9f37615ce52b77bc8505aeba58d284366" Oct 30 08:06:14.432823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:14.432785 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997beedc-265c-4542-b38e-100bdae82e19" path="/var/lib/kubelet/pods/997beedc-265c-4542-b38e-100bdae82e19/volumes" Oct 30 08:06:14.530862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:14.530826 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:16.403126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:16.403078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:18.618449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:18.618404 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:20.464689 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.464649 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248"] Oct 30 08:06:20.465279 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465264 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e21f34cf-b472-4fad-a8c4-b2ee6af46079" containerName="revoke-keys" Oct 30 08:06:20.465326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465279 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e21f34cf-b472-4fad-a8c4-b2ee6af46079" containerName="revoke-keys" Oct 30 08:06:20.465326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465296 5894 cpu_manager.go:414] "containerMap: removing container" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" Oct 30 08:06:20.465326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465302 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" Oct 30 08:06:20.465433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465392 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e21f34cf-b472-4fad-a8c4-b2ee6af46079" containerName="revoke-keys" Oct 30 08:06:20.465433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.465401 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="997beedc-265c-4542-b38e-100bdae82e19" containerName="istio-proxy" Oct 30 08:06:20.469472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.469450 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.498107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.498079 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:06:20.525259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.525232 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:06:20.555947 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.555914 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:06:20.556331 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.556311 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:06:20.558990 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.558961 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248"] Oct 30 08:06:20.559260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.559229 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gkp\" (UniqueName: \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.559460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.559437 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.559552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.559512 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.660896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.660860 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gkp\" (UniqueName: \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.661079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.661024 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.661079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.661069 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.661846 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.661826 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.663579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.663560 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.700312 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.700282 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gkp\" (UniqueName: \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\") pod \"maas-api-revoke-sub-fe25ed43-nl248\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.783992 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.783958 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:20.908563 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:20.907627 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:21.098305 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:21.098269 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da6bcdc_e7ae_4465_baa2_3de77d5b1b47.slice/crio-c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c WatchSource:0}: Error finding container c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c: Status 404 returned error can't find the container with id c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c Oct 30 08:06:21.099334 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.099303 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248"] Oct 30 08:06:21.649887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.649793 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" event={"ID":"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47","Type":"ContainerStarted","Data":"e82f3bd2fa3eec67746ef1a0e223a24d23704c0f1d2193f2128ce560997690b5"} Oct 30 08:06:21.649887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.649847 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" event={"ID":"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47","Type":"ContainerStarted","Data":"c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c"} Oct 30 08:06:21.712278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.712225 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" podStartSLOduration=1.712211588 podStartE2EDuration="1.712211588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:21.710760788 +0000 UTC m=+4145.401678226" watchObservedRunningTime="2026-10-30 08:06:21.712211588 +0000 UTC m=+4145.403129025" Oct 30 08:06:21.856986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.856954 5894 scope.go:122] "RemoveContainer" containerID="2d3f6f6a3eacb43eaa6b6959069db45e9e386541eb3f51e5203919db839c3758" Oct 30 08:06:21.867217 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.867187 5894 scope.go:122] "RemoveContainer" containerID="3d1fe9c5f7ff61959c752b873b024e37b0cbbc95a595ecf78458ba0627e6c5aa" Oct 30 08:06:21.876819 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.876766 5894 scope.go:122] "RemoveContainer" containerID="aedbad0c82d23573ecfdbb12a1d06c03563d21eca6fe4a41398309c7c10549f9" Oct 30 08:06:21.885973 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:21.885951 5894 scope.go:122] "RemoveContainer" containerID="d34b6a0baed90b366cd5f93bb8d8f0d830e18da4535875e9be911038f3b26fd2" Oct 30 08:06:22.658374 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:22.658340 5894 generic.go:356] "Generic (PLEG): container finished" podID="3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" containerID="e82f3bd2fa3eec67746ef1a0e223a24d23704c0f1d2193f2128ce560997690b5" exitCode=0 Oct 30 08:06:22.658856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:22.658426 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" event={"ID":"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47","Type":"ContainerDied","Data":"e82f3bd2fa3eec67746ef1a0e223a24d23704c0f1d2193f2128ce560997690b5"} Oct 30 08:06:23.247496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.247460 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:23.422229 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.422196 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:23.823579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.823553 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:23.951547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.951462 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\") pod \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " Oct 30 08:06:23.951713 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.951593 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\") pod \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " Oct 30 08:06:23.951776 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.951763 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\" (UniqueName: \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\") pod \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\" (UID: \"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47\") " Oct 30 08:06:23.951999 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.951974 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca" pod "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" (UID: "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:23.952143 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.952129 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:23.953802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.953780 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca" pod "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" (UID: "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:23.953911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:23.953853 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp" pod "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" (UID: "3da6bcdc-e7ae-4465-baa2-3de77d5b1b47"). InnerVolumeSpecName "kube-api-access-v4gkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:24.053886 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:24.053844 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gkp\" (UniqueName: \"kubernetes.io/projected/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-kube-api-access-v4gkp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:24.053886 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:24.053877 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da6bcdc-e7ae-4465-baa2-3de77d5b1b47-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:24.674602 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:24.674570 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" Oct 30 08:06:24.674817 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:24.674568 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe25ed43-nl248" event={"ID":"3da6bcdc-e7ae-4465-baa2-3de77d5b1b47","Type":"ContainerDied","Data":"c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c"} Oct 30 08:06:24.674817 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:24.674709 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4edb18a80e6bc10e5a5ac9e1e28c6c2ada73f60c7dc17447650b6dcfee7f18c" Oct 30 08:06:25.238263 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:25.238228 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:27.272496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:27.272412 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:29.238506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:29.238469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:31.612256 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.612218 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:31.622653 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.622600 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:06:31.826542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.826490 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:31.961320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.961221 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:31.962034 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.962010 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" containerName="revoke-keys" Oct 30 08:06:31.962034 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.962034 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" containerName="revoke-keys" Oct 30 08:06:31.962234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.962218 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da6bcdc-e7ae-4465-baa2-3de77d5b1b47" containerName="revoke-keys" Oct 30 08:06:31.966415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:31.966392 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.015861 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.015824 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989104-openshift-default-dockercfg-59vg7\"" Oct 30 08:06:32.028271 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.028238 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:32.065370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065330 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjl8\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065387 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065423 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065470 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065492 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065534 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065601 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065626 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.065857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.065660 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.166669 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.166614 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.166853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.166734 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjl8\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.166853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.166795 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.166853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.166836 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.166997 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167041 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167162 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167116 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167201 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167235 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167256 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167387 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167561 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167662 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167583 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.167923 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.167887 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.169433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.169416 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.169614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.169595 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.203619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.203584 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.232126 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.232055 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjl8\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\") pod \"e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.284330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.284289 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:32.663907 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:32.663869 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0da2b7_27f6_42db_b986_ef69557b394e.slice/crio-6b7e8014163e73d204d2301895785d2395bf87b80751d7b0b2cbaef8bdfc18d0 WatchSource:0}: Error finding container 6b7e8014163e73d204d2301895785d2395bf87b80751d7b0b2cbaef8bdfc18d0: Status 404 returned error can't find the container with id 6b7e8014163e73d204d2301895785d2395bf87b80751d7b0b2cbaef8bdfc18d0 Oct 30 08:06:32.664864 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.664840 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:32.666595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.666563 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.666679 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.666628 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.666723 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.666688 5894 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 08:06:32.770878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.770837 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" event={"ID":"df0da2b7-27f6-42db-b986-ef69557b394e","Type":"ContainerStarted","Data":"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1"} Oct 30 08:06:32.771071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.770905 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" event={"ID":"df0da2b7-27f6-42db-b986-ef69557b394e","Type":"ContainerStarted","Data":"6b7e8014163e73d204d2301895785d2395bf87b80751d7b0b2cbaef8bdfc18d0"} Oct 30 08:06:32.840841 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:32.840785 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podStartSLOduration=1.84077129 podStartE2EDuration="1.84077129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:32.837316815 +0000 UTC m=+4156.528234253" watchObservedRunningTime="2026-10-30 08:06:32.84077129 +0000 UTC m=+4156.531688787" Oct 30 08:06:33.289257 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:33.289217 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:33.289627 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:33.289601 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:33.289833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:33.289809 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:33.850285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:33.850250 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:06:33.888874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:33.888761 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:34.109278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.109186 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:34.247332 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.247298 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4"] Oct 30 08:06:34.254342 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.253658 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.303109 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.303902 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.303974 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.304071 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.305080 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:34.305514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.305154 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:34.320655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.320302 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4"] Oct 30 08:06:34.427717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.424729 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.427717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.424826 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqznb\" (UniqueName: \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.427717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.424883 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.525974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.525933 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.526170 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.526019 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqznb\" (UniqueName: \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.526170 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.526058 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.528010 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.527981 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.534472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.534444 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.555848 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.555806 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqznb\" (UniqueName: \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\") pod \"maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.636658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:34.627393 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:34.887987 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:06:34.886173 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:06:35.055949 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.055887 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4"] Oct 30 08:06:35.077065 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.077032 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:35.226180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.226131 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:35.289657 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.289027 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:35.289657 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.289088 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:35.297941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.297821 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x"] Oct 30 08:06:35.301182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.301152 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989104-kzr5x"] Oct 30 08:06:35.358672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.355084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:35.785451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.785418 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:35.848902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.848748 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" event={"ID":"6da48a7f-8a41-42f0-a1f3-244f92c39cf2","Type":"ContainerStarted","Data":"566cf1ff3bce38d07dc9dfc706874bd98b4be1ea4dd52f266ab3d81db988e38a"} Oct 30 08:06:35.848902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.848811 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" event={"ID":"6da48a7f-8a41-42f0-a1f3-244f92c39cf2","Type":"ContainerStarted","Data":"aaba7d08077c97d2f5d256920c799ac3d71fa24c43b0c378043f5655d0d4c36c"} Oct 30 08:06:35.898922 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:35.898861 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" podStartSLOduration=1.898845016 podStartE2EDuration="1.898845016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:35.898723187 +0000 UTC m=+4159.589640627" watchObservedRunningTime="2026-10-30 08:06:35.898845016 +0000 UTC m=+4159.589762452" Oct 30 08:06:36.290537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.288757 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:36.290537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.288827 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:36.398686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.394900 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:36.434944 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.434900 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce7d0301-446a-44e6-b234-a77e099bdeb5" path="/var/lib/kubelet/pods/ce7d0301-446a-44e6-b234-a77e099bdeb5/volumes" Oct 30 08:06:36.543159 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.543071 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:36.858317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.858226 5894 generic.go:356] "Generic (PLEG): container finished" podID="6da48a7f-8a41-42f0-a1f3-244f92c39cf2" containerID="566cf1ff3bce38d07dc9dfc706874bd98b4be1ea4dd52f266ab3d81db988e38a" exitCode=0 Oct 30 08:06:36.858490 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:36.858426 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" event={"ID":"6da48a7f-8a41-42f0-a1f3-244f92c39cf2","Type":"ContainerDied","Data":"566cf1ff3bce38d07dc9dfc706874bd98b4be1ea4dd52f266ab3d81db988e38a"} Oct 30 08:06:37.292658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:37.290070 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:37.292658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:37.290140 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:37.525212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:37.525179 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:38.108086 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.107611 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:38.202710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.199167 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\" (UniqueName: \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\") pod \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " Oct 30 08:06:38.202710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.201948 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\") pod \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " Oct 30 08:06:38.202710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.202048 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\") pod \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\" (UID: \"6da48a7f-8a41-42f0-a1f3-244f92c39cf2\") " Oct 30 08:06:38.202982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.202886 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca" pod "6da48a7f-8a41-42f0-a1f3-244f92c39cf2" (UID: "6da48a7f-8a41-42f0-a1f3-244f92c39cf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:38.203713 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.203665 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb" pod "6da48a7f-8a41-42f0-a1f3-244f92c39cf2" (UID: "6da48a7f-8a41-42f0-a1f3-244f92c39cf2"). InnerVolumeSpecName "kube-api-access-jqznb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:38.205058 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.205032 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca" pod "6da48a7f-8a41-42f0-a1f3-244f92c39cf2" (UID: "6da48a7f-8a41-42f0-a1f3-244f92c39cf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:38.296729 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.296154 5894 patch_prober.go:29] interesting pod/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" start-of-body= Oct 30 08:06:38.296729 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.296224 5894 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.160:15021/healthz/ready\": dial tcp 10.134.1.160:15021: connect: connection refused" Oct 30 08:06:38.303984 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.303949 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqznb\" (UniqueName: \"kubernetes.io/projected/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-kube-api-access-jqznb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:38.303984 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.303978 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:38.303984 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.303987 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da48a7f-8a41-42f0-a1f3-244f92c39cf2-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:38.876578 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.876542 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" Oct 30 08:06:38.876777 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.876574 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4" event={"ID":"6da48a7f-8a41-42f0-a1f3-244f92c39cf2","Type":"ContainerDied","Data":"aaba7d08077c97d2f5d256920c799ac3d71fa24c43b0c378043f5655d0d4c36c"} Oct 30 08:06:38.876777 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:38.876626 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaba7d08077c97d2f5d256920c799ac3d71fa24c43b0c378043f5655d0d4c36c" Oct 30 08:06:39.290134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:39.290097 5894 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:39.290678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:39.290606 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:39.291514 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:39.291492 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:39.487146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:39.487108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:41.372441 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.372400 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:41.732285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.732207 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf"] Oct 30 08:06:41.732830 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.732816 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6da48a7f-8a41-42f0-a1f3-244f92c39cf2" containerName="revoke-keys" Oct 30 08:06:41.732891 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.732831 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da48a7f-8a41-42f0-a1f3-244f92c39cf2" containerName="revoke-keys" Oct 30 08:06:41.732951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.732942 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da48a7f-8a41-42f0-a1f3-244f92c39cf2" containerName="revoke-keys" Oct 30 08:06:41.736737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.736720 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.757113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.757076 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf"] Oct 30 08:06:41.768264 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.768224 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:06:41.768264 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.768249 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:06:41.768799 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.768224 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:06:41.768799 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.768305 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:06:41.847457 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.847420 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.847718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.847502 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.847803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.847759 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsb9\" (UniqueName: \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.948525 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.948485 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.948766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.948603 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsb9\" (UniqueName: \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.948766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.948713 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.949227 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.949202 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.951122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.951101 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:41.974406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:41.974373 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsb9\" (UniqueName: \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\") pod \"maas-api-revoke-sub-582beb14-l8vlf\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:42.050788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.050757 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:42.160277 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.160166 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:42.279421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.279379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:42.363189 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:42.363154 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932c3733_e64e_4980_ac4a_c87bb7ab593a.slice/crio-6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175 WatchSource:0}: Error finding container 6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175: Status 404 returned error can't find the container with id 6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175 Oct 30 08:06:42.363855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.363830 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf"] Oct 30 08:06:42.458556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.458521 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:42.458995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.458956 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" containerID="cri-o://b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1" gracePeriod=30 Oct 30 08:06:42.520440 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.520392 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:42.907690 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.907623 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" event={"ID":"932c3733-e64e-4980-ac4a-c87bb7ab593a","Type":"ContainerStarted","Data":"1b8ef23eea2b3e0c28ccb27cf597cff83282e1936c2b591b669eed468e2832e0"} Oct 30 08:06:42.907879 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.907709 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" event={"ID":"932c3733-e64e-4980-ac4a-c87bb7ab593a","Type":"ContainerStarted","Data":"6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175"} Oct 30 08:06:42.953786 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:42.953729 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" podStartSLOduration=1.953713954 podStartE2EDuration="1.953713954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:06:42.952502618 +0000 UTC m=+4166.643420056" watchObservedRunningTime="2026-10-30 08:06:42.953713954 +0000 UTC m=+4166.644631435" Oct 30 08:06:43.314224 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:43.314184 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:43.916790 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:43.916755 5894 generic.go:356] "Generic (PLEG): container finished" podID="932c3733-e64e-4980-ac4a-c87bb7ab593a" containerID="1b8ef23eea2b3e0c28ccb27cf597cff83282e1936c2b591b669eed468e2832e0" exitCode=0 Oct 30 08:06:43.917238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:43.916878 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" event={"ID":"932c3733-e64e-4980-ac4a-c87bb7ab593a","Type":"ContainerDied","Data":"1b8ef23eea2b3e0c28ccb27cf597cff83282e1936c2b591b669eed468e2832e0"} Oct 30 08:06:44.713175 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:44.713134 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 08:06:44.749703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:44.749659 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 08:06:44.874851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:44.874817 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:45.111222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.111193 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:45.195404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.195373 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\") pod \"932c3733-e64e-4980-ac4a-c87bb7ab593a\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " Oct 30 08:06:45.195609 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.195526 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\") pod \"932c3733-e64e-4980-ac4a-c87bb7ab593a\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " Oct 30 08:06:45.195609 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.195551 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\" (UniqueName: \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\") pod \"932c3733-e64e-4980-ac4a-c87bb7ab593a\" (UID: \"932c3733-e64e-4980-ac4a-c87bb7ab593a\") " Oct 30 08:06:45.195843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.195818 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca" pod "932c3733-e64e-4980-ac4a-c87bb7ab593a" (UID: "932c3733-e64e-4980-ac4a-c87bb7ab593a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:45.196008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.195989 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:45.197758 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.197730 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca" pod "932c3733-e64e-4980-ac4a-c87bb7ab593a" (UID: "932c3733-e64e-4980-ac4a-c87bb7ab593a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:45.197861 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.197802 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9" pod "932c3733-e64e-4980-ac4a-c87bb7ab593a" (UID: "932c3733-e64e-4980-ac4a-c87bb7ab593a"). InnerVolumeSpecName "kube-api-access-vjsb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:45.297100 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.297059 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/932c3733-e64e-4980-ac4a-c87bb7ab593a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:45.297100 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.297099 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjsb9\" (UniqueName: \"kubernetes.io/projected/932c3733-e64e-4980-ac4a-c87bb7ab593a-kube-api-access-vjsb9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:45.852522 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.852476 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:45.932688 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.932621 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" Oct 30 08:06:45.932877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.932659 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-582beb14-l8vlf" event={"ID":"932c3733-e64e-4980-ac4a-c87bb7ab593a","Type":"ContainerDied","Data":"6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175"} Oct 30 08:06:45.932877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:45.932779 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b6bb66456a4bab1bd20aeb2bc6c16c9fde40b6dab5437d443744613b77f9175" Oct 30 08:06:46.643368 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.643330 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:06:46.647349 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.647316 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" podUID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" containerName="main" containerID="cri-o://55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f" gracePeriod=45 Oct 30 08:06:46.925774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.925746 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:06:46.942083 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.942051 5894 generic.go:356] "Generic (PLEG): container finished" podID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" containerID="55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f" exitCode=0 Oct 30 08:06:46.942238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.942095 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" Oct 30 08:06:46.942238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.942096 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" event={"ID":"8178f323-24f3-42ad-be83-1fd8d0edd7d7","Type":"ContainerDied","Data":"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f"} Oct 30 08:06:46.942238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.942217 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw" event={"ID":"8178f323-24f3-42ad-be83-1fd8d0edd7d7","Type":"ContainerDied","Data":"d84666487c793137edfdbc2af80316c694c85bc3302a8acd78bc54d9d21702b3"} Oct 30 08:06:46.942369 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.942244 5894 scope.go:122] "RemoveContainer" containerID="55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f" Oct 30 08:06:46.952659 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.952604 5894 scope.go:122] "RemoveContainer" containerID="55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f" Oct 30 08:06:46.952988 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:06:46.952963 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f\": container with ID starting with 55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f not found: ID does not exist" containerID="55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f" Oct 30 08:06:46.953038 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:46.952998 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f"} err="failed to get container status \"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f\": rpc error: code = NotFound desc = could not find container \"55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f\": container with ID starting with 55c39a4ecbe95d9db03e41d8e0045b0d5646d502d1c311c6cdf75fecd963c38f not found: ID does not exist" Oct 30 08:06:47.024919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.024866 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025081 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.024963 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025081 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025040 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025118 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\" (UniqueName: \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025163 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025210 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\") pod \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\" (UID: \"8178f323-24f3-42ad-be83-1fd8d0edd7d7\") " Oct 30 08:06:47.025408 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025263 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.025408 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025299 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.025533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025451 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.025826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025805 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.025932 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025829 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.025932 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.025842 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.027233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.027206 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.027351 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.027308 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:47.027411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.027364 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf" pod "8178f323-24f3-42ad-be83-1fd8d0edd7d7" (UID: "8178f323-24f3-42ad-be83-1fd8d0edd7d7"). InnerVolumeSpecName "kube-api-access-nwldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.127375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.127340 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwldf\" (UniqueName: \"kubernetes.io/projected/8178f323-24f3-42ad-be83-1fd8d0edd7d7-kube-api-access-nwldf\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.127375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.127373 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8178f323-24f3-42ad-be83-1fd8d0edd7d7-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.127375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.127383 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8178f323-24f3-42ad-be83-1fd8d0edd7d7-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.176816 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.176716 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:47.315453 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.315404 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:06:47.318792 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.318757 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-d89593/rate-test-model-d89593-kserve-5864855677-n7mgw"] Oct 30 08:06:47.745360 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.745294 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:47.837605 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837572 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837673 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837704 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837730 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837830 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837856 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837881 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837918 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.837972 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838018 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838058 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\") pod \"df0da2b7-27f6-42db-b986-ef69557b394e\" (UID: \"df0da2b7-27f6-42db-b986-ef69557b394e\") " Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838068 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838542 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838558 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.838622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838568 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.839578 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.838949 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.839578 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.839044 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:47.840499 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.840466 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.841073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.841029 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:06:47.841073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.841065 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:47.841221 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.841166 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8" pod "df0da2b7-27f6-42db-b986-ef69557b394e" (UID: "df0da2b7-27f6-42db-b986-ef69557b394e"). InnerVolumeSpecName "kube-api-access-wsjl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:47.939580 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939541 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df0da2b7-27f6-42db-b986-ef69557b394e-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.939580 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939572 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.939580 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939584 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.939869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939597 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjl8\" (UniqueName: \"kubernetes.io/projected/df0da2b7-27f6-42db-b986-ef69557b394e-kube-api-access-wsjl8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.939869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939612 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df0da2b7-27f6-42db-b986-ef69557b394e-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.939869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.939620 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df0da2b7-27f6-42db-b986-ef69557b394e-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:47.952153 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.952115 5894 generic.go:356] "Generic (PLEG): container finished" podID="df0da2b7-27f6-42db-b986-ef69557b394e" containerID="b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1" exitCode=0 Oct 30 08:06:47.952356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.952162 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" Oct 30 08:06:47.952356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.952174 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" event={"ID":"df0da2b7-27f6-42db-b986-ef69557b394e","Type":"ContainerDied","Data":"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1"} Oct 30 08:06:47.952356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.952225 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks" event={"ID":"df0da2b7-27f6-42db-b986-ef69557b394e","Type":"ContainerDied","Data":"6b7e8014163e73d204d2301895785d2395bf87b80751d7b0b2cbaef8bdfc18d0"} Oct 30 08:06:47.952356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.952256 5894 scope.go:122] "RemoveContainer" containerID="b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1" Oct 30 08:06:47.965611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.965576 5894 scope.go:122] "RemoveContainer" containerID="b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1" Oct 30 08:06:47.966037 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:06:47.966012 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1\": container with ID starting with b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1 not found: ID does not exist" containerID="b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1" Oct 30 08:06:47.966139 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:47.966044 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1"} err="failed to get container status \"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1\": rpc error: code = NotFound desc = could not find container \"b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1\": container with ID starting with b67eb84187366f13562242fdfdb2b66dc099e4c29fe7c45d70fd93f04208aac1 not found: ID does not exist" Oct 30 08:06:48.017855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:48.017772 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:48.020849 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:48.020821 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989104-openshift-default-7769f5db8c-6jkks"] Oct 30 08:06:48.432284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:48.432246 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" path="/var/lib/kubelet/pods/8178f323-24f3-42ad-be83-1fd8d0edd7d7/volumes" Oct 30 08:06:48.432786 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:48.432771 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" path="/var/lib/kubelet/pods/df0da2b7-27f6-42db-b986-ef69557b394e/volumes" Oct 30 08:06:48.972724 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:48.972676 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" podUID="63b41062-701f-40ff-9c4d-1b28c624e7b6" containerName="main" containerID="cri-o://2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8" gracePeriod=45 Oct 30 08:06:49.063107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.063068 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:49.298833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.298797 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:06:49.353740 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.353709 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.353947 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.353820 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.353947 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.353860 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\" (UniqueName: \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.354079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.353979 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:49.354079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.353995 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.354079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354065 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.354248 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354172 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\") pod \"63b41062-701f-40ff-9c4d-1b28c624e7b6\" (UID: \"63b41062-701f-40ff-9c4d-1b28c624e7b6\") " Oct 30 08:06:49.354472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354446 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:49.354716 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354690 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.354822 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354721 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.354881 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.354846 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:49.356739 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.356699 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "kube-api-access-9lvs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:49.356900 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.356880 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:06:49.357017 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.356992 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs" pod "63b41062-701f-40ff-9c4d-1b28c624e7b6" (UID: "63b41062-701f-40ff-9c4d-1b28c624e7b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:49.456129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.456094 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63b41062-701f-40ff-9c4d-1b28c624e7b6-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.456129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.456124 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.456129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.456134 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lvs9\" (UniqueName: \"kubernetes.io/projected/63b41062-701f-40ff-9c4d-1b28c624e7b6-kube-api-access-9lvs9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.456390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.456143 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/63b41062-701f-40ff-9c4d-1b28c624e7b6-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:49.714385 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.714335 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" containerID="cri-o://901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b" gracePeriod=25 Oct 30 08:06:49.749993 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.749952 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" containerID="cri-o://f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d" gracePeriod=25 Oct 30 08:06:49.968911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.968823 5894 generic.go:356] "Generic (PLEG): container finished" podID="63b41062-701f-40ff-9c4d-1b28c624e7b6" containerID="2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8" exitCode=0 Oct 30 08:06:49.968911 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.968871 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" Oct 30 08:06:49.969149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.968904 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" event={"ID":"63b41062-701f-40ff-9c4d-1b28c624e7b6","Type":"ContainerDied","Data":"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8"} Oct 30 08:06:49.969149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.968958 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr" event={"ID":"63b41062-701f-40ff-9c4d-1b28c624e7b6","Type":"ContainerDied","Data":"21721b3ee29185abead82e4ebd49a8401a6ea11fecbdfdaa30be14b36ee8f6a6"} Oct 30 08:06:49.969149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.968981 5894 scope.go:122] "RemoveContainer" containerID="2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8" Oct 30 08:06:49.980449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.980195 5894 scope.go:122] "RemoveContainer" containerID="2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8" Oct 30 08:06:49.980830 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:06:49.980500 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8\": container with ID starting with 2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8 not found: ID does not exist" containerID="2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8" Oct 30 08:06:49.980830 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:49.980529 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8"} err="failed to get container status \"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8\": rpc error: code = NotFound desc = could not find container \"2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8\": container with ID starting with 2fca8a119cd9fc3217df0fad20bd72d7108cb91b214b4220da9582e7c487e6e8 not found: ID does not exist" Oct 30 08:06:50.029301 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.029266 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:06:50.044327 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.044281 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-054ef1/rate-test-model-054ef1-kserve-7b746d874f-x72pr"] Oct 30 08:06:50.176208 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176175 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr"] Oct 30 08:06:50.176886 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176868 5894 cpu_manager.go:414] "containerMap: removing container" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" Oct 30 08:06:50.176950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176889 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" Oct 30 08:06:50.176950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176924 5894 cpu_manager.go:414] "containerMap: removing container" podUID="63b41062-701f-40ff-9c4d-1b28c624e7b6" containerName="main" Oct 30 08:06:50.176950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176930 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b41062-701f-40ff-9c4d-1b28c624e7b6" containerName="main" Oct 30 08:06:50.176950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176940 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" containerName="main" Oct 30 08:06:50.176950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176946 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" containerName="main" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176960 5894 cpu_manager.go:414] "containerMap: removing container" podUID="932c3733-e64e-4980-ac4a-c87bb7ab593a" containerName="revoke-keys" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.176966 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932c3733-e64e-4980-ac4a-c87bb7ab593a" containerName="revoke-keys" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.177052 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="df0da2b7-27f6-42db-b986-ef69557b394e" containerName="istio-proxy" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.177064 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b41062-701f-40ff-9c4d-1b28c624e7b6" containerName="main" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.177073 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8178f323-24f3-42ad-be83-1fd8d0edd7d7" containerName="main" Oct 30 08:06:50.177122 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.177081 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="932c3733-e64e-4980-ac4a-c87bb7ab593a" containerName="revoke-keys" Oct 30 08:06:50.181818 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.181791 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.200498 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.200463 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:06:50.200700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.200509 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:06:50.200700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.200551 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:06:50.200700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.200628 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:06:50.201903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.201881 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr"] Oct 30 08:06:50.267016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.266928 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.267016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.266973 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzfr\" (UniqueName: \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.267016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.267010 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.368220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.368184 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.368220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.368232 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzfr\" (UniqueName: \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.368459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.368343 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.369057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.369029 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.370893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.370876 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.394688 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.394647 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzfr\" (UniqueName: \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\") pod \"maas-api-revoke-sub-ab551f54-hlxvr\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.432240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.432196 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b41062-701f-40ff-9c4d-1b28c624e7b6" path="/var/lib/kubelet/pods/63b41062-701f-40ff-9c4d-1b28c624e7b6/volumes" Oct 30 08:06:50.497222 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.497187 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:50.807452 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:50.807419 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c6c883_8456_403b_909c_e79a3425d19d.slice/crio-641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7 WatchSource:0}: Error finding container 641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7: Status 404 returned error can't find the container with id 641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7 Oct 30 08:06:50.808486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.808455 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr"] Oct 30 08:06:50.978725 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.978681 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" event={"ID":"36c6c883-8456-403b-909c-e79a3425d19d","Type":"ContainerStarted","Data":"641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7"} Oct 30 08:06:50.994046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:50.994010 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:51.989955 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:51.989852 5894 generic.go:356] "Generic (PLEG): container finished" podID="36c6c883-8456-403b-909c-e79a3425d19d" containerID="58f1d10b58b2b58660112d3661e240eeb235e1601d1df23fe3c9520681b63289" exitCode=0 Oct 30 08:06:51.990151 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:51.989945 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" event={"ID":"36c6c883-8456-403b-909c-e79a3425d19d","Type":"ContainerDied","Data":"58f1d10b58b2b58660112d3661e240eeb235e1601d1df23fe3c9520681b63289"} Oct 30 08:06:52.164396 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.164362 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs"] Oct 30 08:06:52.169100 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.169077 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.189589 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.189558 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs"] Oct 30 08:06:52.291486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.291449 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.291744 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.291626 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nj4r\" (UniqueName: \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.291824 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.291767 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.393452 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.393414 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nj4r\" (UniqueName: \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.393611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.393476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.393611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.393576 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.394242 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.394210 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.396038 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.396013 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.420201 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.420171 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nj4r\" (UniqueName: \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\") pod \"maas-api-revoke-sub-2874b07e-p5vjs\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.486188 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.486156 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:52.701099 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.701056 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:06:52.701099 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.701079 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:06:52.794998 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:06:52.794961 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5eb796_706c_4cbb_b3a3_8a91c14fa315.slice/crio-3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369 WatchSource:0}: Error finding container 3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369: Status 404 returned error can't find the container with id 3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369 Oct 30 08:06:52.796327 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.796301 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs"] Oct 30 08:06:52.912071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.912041 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:52.999248 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:52.999210 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" event={"ID":"8a5eb796-706c-4cbb-b3a3-8a91c14fa315","Type":"ContainerStarted","Data":"3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369"} Oct 30 08:06:53.181093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.181067 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:53.304513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.304477 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\" (UniqueName: \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\") pod \"36c6c883-8456-403b-909c-e79a3425d19d\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " Oct 30 08:06:53.304708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.304681 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\") pod \"36c6c883-8456-403b-909c-e79a3425d19d\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " Oct 30 08:06:53.304841 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.304824 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\") pod \"36c6c883-8456-403b-909c-e79a3425d19d\" (UID: \"36c6c883-8456-403b-909c-e79a3425d19d\") " Oct 30 08:06:53.305155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.305128 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca" pod "36c6c883-8456-403b-909c-e79a3425d19d" (UID: "36c6c883-8456-403b-909c-e79a3425d19d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:53.305478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.305281 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:53.306736 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.306701 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr" pod "36c6c883-8456-403b-909c-e79a3425d19d" (UID: "36c6c883-8456-403b-909c-e79a3425d19d"). InnerVolumeSpecName "kube-api-access-gtzfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:53.306901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.306878 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca" pod "36c6c883-8456-403b-909c-e79a3425d19d" (UID: "36c6c883-8456-403b-909c-e79a3425d19d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:53.406173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.406073 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtzfr\" (UniqueName: \"kubernetes.io/projected/36c6c883-8456-403b-909c-e79a3425d19d-kube-api-access-gtzfr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:53.406173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:53.406102 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36c6c883-8456-403b-909c-e79a3425d19d-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:54.009321 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.009231 5894 generic.go:356] "Generic (PLEG): container finished" podID="8a5eb796-706c-4cbb-b3a3-8a91c14fa315" containerID="3290b7813aa3e9db3a78846a38965fb11dd9befa272dc26cdbd926bab6296fef" exitCode=0 Oct 30 08:06:54.009496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.009317 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" event={"ID":"8a5eb796-706c-4cbb-b3a3-8a91c14fa315","Type":"ContainerDied","Data":"3290b7813aa3e9db3a78846a38965fb11dd9befa272dc26cdbd926bab6296fef"} Oct 30 08:06:54.010856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.010835 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" Oct 30 08:06:54.010856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.010847 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab551f54-hlxvr" event={"ID":"36c6c883-8456-403b-909c-e79a3425d19d","Type":"ContainerDied","Data":"641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7"} Oct 30 08:06:54.011054 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.010883 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="641fb67b8050e72472015aab9dbf21f1c7f14fb9b1ce9995c1173da9d43b19b7" Oct 30 08:06:54.756996 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:54.756959 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:55.179919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.179887 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:55.328316 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.328279 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\" (UniqueName: \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\") pod \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " Oct 30 08:06:55.328537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.328370 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\") pod \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " Oct 30 08:06:55.328537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.328404 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\") pod \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\" (UID: \"8a5eb796-706c-4cbb-b3a3-8a91c14fa315\") " Oct 30 08:06:55.328903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.328877 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca" pod "8a5eb796-706c-4cbb-b3a3-8a91c14fa315" (UID: "8a5eb796-706c-4cbb-b3a3-8a91c14fa315"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:06:55.330596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.330565 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca" pod "8a5eb796-706c-4cbb-b3a3-8a91c14fa315" (UID: "8a5eb796-706c-4cbb-b3a3-8a91c14fa315"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:06:55.330727 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.330617 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r" pod "8a5eb796-706c-4cbb-b3a3-8a91c14fa315" (UID: "8a5eb796-706c-4cbb-b3a3-8a91c14fa315"). InnerVolumeSpecName "kube-api-access-4nj4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:06:55.430233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.430196 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nj4r\" (UniqueName: \"kubernetes.io/projected/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-kube-api-access-4nj4r\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:55.430233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.430231 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:55.430233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:55.430241 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5eb796-706c-4cbb-b3a3-8a91c14fa315-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:06:56.027896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:56.027860 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" Oct 30 08:06:56.027896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:56.027880 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2874b07e-p5vjs" event={"ID":"8a5eb796-706c-4cbb-b3a3-8a91c14fa315","Type":"ContainerDied","Data":"3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369"} Oct 30 08:06:56.028455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:56.027931 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fa76f3a7b419a9b101db8bbfee2ddadff7917d7894b11c3aad4dca666128369" Oct 30 08:06:56.790710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:56.790667 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:58.415517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:58.415468 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:59.117359 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:59.117289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:59.254419 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:59.254374 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:06:59.255782 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:59.255760 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4"] Oct 30 08:06:59.284891 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:59.284845 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6051b2-w8db4"] Oct 30 08:06:59.730615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:06:59.730574 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:00.431838 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.431806 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da48a7f-8a41-42f0-a1f3-244f92c39cf2" path="/var/lib/kubelet/pods/6da48a7f-8a41-42f0-a1f3-244f92c39cf2/volumes" Oct 30 08:07:00.509200 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509166 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd"] Oct 30 08:07:00.509810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509796 5894 cpu_manager.go:414] "containerMap: removing container" podUID="8a5eb796-706c-4cbb-b3a3-8a91c14fa315" containerName="revoke-keys" Oct 30 08:07:00.509871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509812 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5eb796-706c-4cbb-b3a3-8a91c14fa315" containerName="revoke-keys" Oct 30 08:07:00.509871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509837 5894 cpu_manager.go:414] "containerMap: removing container" podUID="36c6c883-8456-403b-909c-e79a3425d19d" containerName="revoke-keys" Oct 30 08:07:00.509871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509842 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36c6c883-8456-403b-909c-e79a3425d19d" containerName="revoke-keys" Oct 30 08:07:00.509971 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509930 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="36c6c883-8456-403b-909c-e79a3425d19d" containerName="revoke-keys" Oct 30 08:07:00.509971 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.509946 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5eb796-706c-4cbb-b3a3-8a91c14fa315" containerName="revoke-keys" Oct 30 08:07:00.513722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.513704 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.531367 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.531335 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:07:00.546016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.545972 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:07:00.546217 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.546032 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:07:00.546217 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.546043 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:07:00.549235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.549207 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd"] Oct 30 08:07:00.704238 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.704153 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.704407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.704338 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96r9s\" (UniqueName: \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.704462 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.704448 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.809975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.809931 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.812197 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.811751 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.812623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.812591 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96r9s\" (UniqueName: \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.812811 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.812774 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.815890 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.815834 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:00.851376 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:00.851306 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96r9s\" (UniqueName: \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\") pod \"maas-api-revoke-sub-7f55da64-2kdrd\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:01.127703 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:01.127627 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:01.195969 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:01.195871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:01.525422 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:01.525391 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd"] Oct 30 08:07:01.525530 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:01.525492 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5353cb5c_8121_4ba8_9db7_a82249758e41.slice/crio-cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3 WatchSource:0}: Error finding container cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3: Status 404 returned error can't find the container with id cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3 Oct 30 08:07:02.078170 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:02.078076 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" event={"ID":"5353cb5c-8121-4ba8-9db7-a82249758e41","Type":"ContainerStarted","Data":"30fb7a450419b1373d6c2e077d836870d303149549c999d6cbfa73ef48aa370a"} Oct 30 08:07:02.078170 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:02.078141 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" event={"ID":"5353cb5c-8121-4ba8-9db7-a82249758e41","Type":"ContainerStarted","Data":"cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3"} Oct 30 08:07:02.133053 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:02.132994 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" podStartSLOduration=2.132979797 podStartE2EDuration="2.132979797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:02.128228393 +0000 UTC m=+4185.819145831" watchObservedRunningTime="2026-10-30 08:07:02.132979797 +0000 UTC m=+4185.823897233" Oct 30 08:07:02.700243 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:02.700146 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:02.700243 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:02.700190 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:03.089026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.088993 5894 generic.go:356] "Generic (PLEG): container finished" podID="5353cb5c-8121-4ba8-9db7-a82249758e41" containerID="30fb7a450419b1373d6c2e077d836870d303149549c999d6cbfa73ef48aa370a" exitCode=0 Oct 30 08:07:03.089478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.089078 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" event={"ID":"5353cb5c-8121-4ba8-9db7-a82249758e41","Type":"ContainerDied","Data":"30fb7a450419b1373d6c2e077d836870d303149549c999d6cbfa73ef48aa370a"} Oct 30 08:07:03.107747 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.107716 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q"] Oct 30 08:07:03.112198 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.112172 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.131754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.131720 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q"] Oct 30 08:07:03.143546 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.143508 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.143744 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.143562 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.143801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.143765 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98d9\" (UniqueName: \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.244784 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.244743 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98d9\" (UniqueName: \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.245057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.244958 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.245057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.244996 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.245600 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.245566 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.247511 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.247483 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.270999 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.270963 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98d9\" (UniqueName: \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\") pod \"maas-api-revoke-sub-c0ebcc12-qrz2q\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.345019 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.344931 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:03.427020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.426984 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:03.730005 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:03.729972 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2179221_3889_471b_929e_0fac6409a40b.slice/crio-4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c WatchSource:0}: Error finding container 4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c: Status 404 returned error can't find the container with id 4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c Oct 30 08:07:03.731366 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:03.731341 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q"] Oct 30 08:07:04.098673 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.098611 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" event={"ID":"e2179221-3889-471b-929e-0fac6409a40b","Type":"ContainerStarted","Data":"dfb23c635efc201a54f9672dea202367bbc96ed4e864a99ab27a850ab4b08df1"} Oct 30 08:07:04.099129 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.098692 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" event={"ID":"e2179221-3889-471b-929e-0fac6409a40b","Type":"ContainerStarted","Data":"4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c"} Oct 30 08:07:04.146233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.146178 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" podStartSLOduration=1.146158534 podStartE2EDuration="1.146158534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:04.143419799 +0000 UTC m=+4187.834337453" watchObservedRunningTime="2026-10-30 08:07:04.146158534 +0000 UTC m=+4187.837075972" Oct 30 08:07:04.233852 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.233786 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:04.260321 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.260291 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\") pod \"5353cb5c-8121-4ba8-9db7-a82249758e41\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " Oct 30 08:07:04.260484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.260363 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\" (UniqueName: \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\") pod \"5353cb5c-8121-4ba8-9db7-a82249758e41\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " Oct 30 08:07:04.260484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.260406 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\") pod \"5353cb5c-8121-4ba8-9db7-a82249758e41\" (UID: \"5353cb5c-8121-4ba8-9db7-a82249758e41\") " Oct 30 08:07:04.260746 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.260712 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca" pod "5353cb5c-8121-4ba8-9db7-a82249758e41" (UID: "5353cb5c-8121-4ba8-9db7-a82249758e41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:04.260942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.260916 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:04.262616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.262592 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca" pod "5353cb5c-8121-4ba8-9db7-a82249758e41" (UID: "5353cb5c-8121-4ba8-9db7-a82249758e41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:04.262778 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.262755 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s" pod "5353cb5c-8121-4ba8-9db7-a82249758e41" (UID: "5353cb5c-8121-4ba8-9db7-a82249758e41"). InnerVolumeSpecName "kube-api-access-96r9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:04.361965 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.361926 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96r9s\" (UniqueName: \"kubernetes.io/projected/5353cb5c-8121-4ba8-9db7-a82249758e41-kube-api-access-96r9s\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:04.361965 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:04.361959 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5353cb5c-8121-4ba8-9db7-a82249758e41-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:04.829102 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:04.829015 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:07:05.108001 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.107920 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" Oct 30 08:07:05.108001 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.107944 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f55da64-2kdrd" event={"ID":"5353cb5c-8121-4ba8-9db7-a82249758e41","Type":"ContainerDied","Data":"cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3"} Oct 30 08:07:05.108582 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.108006 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdc9356ee8dca9087873ba5074897b93d4a514b014fc75c4ef967a46c20eefa3" Oct 30 08:07:05.110017 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.109982 5894 generic.go:356] "Generic (PLEG): container finished" podID="e2179221-3889-471b-929e-0fac6409a40b" containerID="dfb23c635efc201a54f9672dea202367bbc96ed4e864a99ab27a850ab4b08df1" exitCode=0 Oct 30 08:07:05.110151 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.110022 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" event={"ID":"e2179221-3889-471b-929e-0fac6409a40b","Type":"ContainerDied","Data":"dfb23c635efc201a54f9672dea202367bbc96ed4e864a99ab27a850ab4b08df1"} Oct 30 08:07:05.247075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:05.247036 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:06.278622 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.278595 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:06.388242 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.388159 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\" (UniqueName: \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\") pod \"e2179221-3889-471b-929e-0fac6409a40b\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " Oct 30 08:07:06.388242 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.388231 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\") pod \"e2179221-3889-471b-929e-0fac6409a40b\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " Oct 30 08:07:06.388459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.388322 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\") pod \"e2179221-3889-471b-929e-0fac6409a40b\" (UID: \"e2179221-3889-471b-929e-0fac6409a40b\") " Oct 30 08:07:06.388749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.388721 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca" pod "e2179221-3889-471b-929e-0fac6409a40b" (UID: "e2179221-3889-471b-929e-0fac6409a40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:06.388896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.388878 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2179221-3889-471b-929e-0fac6409a40b-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:06.390424 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.390390 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9" pod "e2179221-3889-471b-929e-0fac6409a40b" (UID: "e2179221-3889-471b-929e-0fac6409a40b"). InnerVolumeSpecName "kube-api-access-x98d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:06.390534 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.390439 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca" pod "e2179221-3889-471b-929e-0fac6409a40b" (UID: "e2179221-3889-471b-929e-0fac6409a40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:06.490531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.490487 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98d9\" (UniqueName: \"kubernetes.io/projected/e2179221-3889-471b-929e-0fac6409a40b-kube-api-access-x98d9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:06.490531 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:06.490523 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2179221-3889-471b-929e-0fac6409a40b-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:07.050396 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.050359 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:07.126626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.126589 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" event={"ID":"e2179221-3889-471b-929e-0fac6409a40b","Type":"ContainerDied","Data":"4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c"} Oct 30 08:07:07.126822 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.126626 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0ebcc12-qrz2q" Oct 30 08:07:07.126822 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.126664 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cdbe064d969e0ded781a890832ed69988108db39baa8bda63bd43faf0b1597c" Oct 30 08:07:07.196898 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.196861 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:07.437415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.437321 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 08:07:07.437926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.437570 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" containerID="cri-o://2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed" gracePeriod=30 Oct 30 08:07:07.481673 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:07.481615 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:08.041849 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.041802 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk"] Oct 30 08:07:08.042495 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042473 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e2179221-3889-471b-929e-0fac6409a40b" containerName="revoke-keys" Oct 30 08:07:08.042611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042496 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2179221-3889-471b-929e-0fac6409a40b" containerName="revoke-keys" Oct 30 08:07:08.042611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042520 5894 cpu_manager.go:414] "containerMap: removing container" podUID="5353cb5c-8121-4ba8-9db7-a82249758e41" containerName="revoke-keys" Oct 30 08:07:08.042611 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042529 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5353cb5c-8121-4ba8-9db7-a82249758e41" containerName="revoke-keys" Oct 30 08:07:08.042817 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042726 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="5353cb5c-8121-4ba8-9db7-a82249758e41" containerName="revoke-keys" Oct 30 08:07:08.042817 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.042760 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2179221-3889-471b-929e-0fac6409a40b" containerName="revoke-keys" Oct 30 08:07:08.046781 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.046759 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.064305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.064271 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:07:08.064305 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.064287 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:07:08.067416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.067388 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk"] Oct 30 08:07:08.111465 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.111428 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2bqm\" (UniqueName: \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.111702 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.111679 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.111805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.111790 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.124940 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.124910 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:07:08.125182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.125161 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:07:08.213293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.213254 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.213515 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.213360 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.213515 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.213391 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2bqm\" (UniqueName: \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.214104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.214078 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.216295 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.216276 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.239939 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.239906 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2bqm\" (UniqueName: \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\") pod \"maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.361667 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.361578 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:08.670449 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:08.670415 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24cc4c3c_6553_4c40_9c5e_72c9d04d7c60.slice/crio-67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659 WatchSource:0}: Error finding container 67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659: Status 404 returned error can't find the container with id 67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659 Oct 30 08:07:08.670887 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:08.670542 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk"] Oct 30 08:07:09.145107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:09.145061 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" event={"ID":"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60","Type":"ContainerStarted","Data":"7575cc0600878048ba274415ba6d8df09fd5a556384b82e8a7503ac305b8934f"} Oct 30 08:07:09.145289 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:09.145145 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" event={"ID":"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60","Type":"ContainerStarted","Data":"67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659"} Oct 30 08:07:09.193042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:09.192979 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" podStartSLOduration=1.192963751 podStartE2EDuration="1.192963751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:09.190233556 +0000 UTC m=+4192.881150996" watchObservedRunningTime="2026-10-30 08:07:09.192963751 +0000 UTC m=+4192.883881188" Oct 30 08:07:09.217314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:09.217278 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:10.018652 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.018591 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 08:07:10.034825 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.034791 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\") pod \"c853d65c-6510-4fc7-addd-a2057dddb2bd\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " Oct 30 08:07:10.035042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.034919 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\" (UniqueName: \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\") pod \"c853d65c-6510-4fc7-addd-a2057dddb2bd\" (UID: \"c853d65c-6510-4fc7-addd-a2057dddb2bd\") " Oct 30 08:07:10.035288 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.035257 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume" pod "c853d65c-6510-4fc7-addd-a2057dddb2bd" (UID: "c853d65c-6510-4fc7-addd-a2057dddb2bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:10.035501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.035456 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c853d65c-6510-4fc7-addd-a2057dddb2bd-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:10.037720 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.037697 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x" pod "c853d65c-6510-4fc7-addd-a2057dddb2bd" (UID: "c853d65c-6510-4fc7-addd-a2057dddb2bd"). InnerVolumeSpecName "kube-api-access-ff79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:10.064147 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.064116 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 08:07:10.136066 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.136035 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\") pod \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " Oct 30 08:07:10.136272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.136246 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\" (UniqueName: \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\") pod \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\" (UID: \"007e648c-d5b5-4b6c-9ece-748e0b79cfd0\") " Oct 30 08:07:10.136475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.136449 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume" pod "007e648c-d5b5-4b6c-9ece-748e0b79cfd0" (UID: "007e648c-d5b5-4b6c-9ece-748e0b79cfd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:10.136727 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.136708 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:10.136805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.136731 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff79x\" (UniqueName: \"kubernetes.io/projected/c853d65c-6510-4fc7-addd-a2057dddb2bd-kube-api-access-ff79x\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:10.138420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.138396 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst" pod "007e648c-d5b5-4b6c-9ece-748e0b79cfd0" (UID: "007e648c-d5b5-4b6c-9ece-748e0b79cfd0"). InnerVolumeSpecName "kube-api-access-n8kst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:10.159150 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.159110 5894 generic.go:356] "Generic (PLEG): container finished" podID="24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" containerID="7575cc0600878048ba274415ba6d8df09fd5a556384b82e8a7503ac305b8934f" exitCode=0 Oct 30 08:07:10.159352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.159190 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" event={"ID":"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60","Type":"ContainerDied","Data":"7575cc0600878048ba274415ba6d8df09fd5a556384b82e8a7503ac305b8934f"} Oct 30 08:07:10.161386 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.161361 5894 generic.go:356] "Generic (PLEG): container finished" podID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerID="901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b" exitCode=0 Oct 30 08:07:10.161544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.161401 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" Oct 30 08:07:10.161544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.161446 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" event={"ID":"c853d65c-6510-4fc7-addd-a2057dddb2bd","Type":"ContainerDied","Data":"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b"} Oct 30 08:07:10.161544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.161514 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt" event={"ID":"c853d65c-6510-4fc7-addd-a2057dddb2bd","Type":"ContainerDied","Data":"c0eb2d4aa720208ab0b1eeb739e0e5a4d100e56b6c494c2486ed44f04c262d5c"} Oct 30 08:07:10.161745 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.161545 5894 scope.go:122] "RemoveContainer" containerID="901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b" Oct 30 08:07:10.163416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.163390 5894 generic.go:356] "Generic (PLEG): container finished" podID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerID="f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d" exitCode=0 Oct 30 08:07:10.163536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.163418 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" event={"ID":"007e648c-d5b5-4b6c-9ece-748e0b79cfd0","Type":"ContainerDied","Data":"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d"} Oct 30 08:07:10.163536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.163451 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" Oct 30 08:07:10.163536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.163472 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd" event={"ID":"007e648c-d5b5-4b6c-9ece-748e0b79cfd0","Type":"ContainerDied","Data":"a7e7a12bf6c3380bc487e3e9ab667840b5caac4f6dac4fa6a2176339ddcda902"} Oct 30 08:07:10.175442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.175421 5894 scope.go:122] "RemoveContainer" containerID="901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b" Oct 30 08:07:10.180148 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:10.180115 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b\": container with ID starting with 901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b not found: ID does not exist" containerID="901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b" Oct 30 08:07:10.180278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.180157 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b"} err="failed to get container status \"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b\": rpc error: code = NotFound desc = could not find container \"901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b\": container with ID starting with 901fd4a98ac4f42991204078cc2170c8c09e1175d7635f17eff42a9fbe56c04b not found: ID does not exist" Oct 30 08:07:10.180278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.180183 5894 scope.go:122] "RemoveContainer" containerID="f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d" Oct 30 08:07:10.191508 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.191473 5894 scope.go:122] "RemoveContainer" containerID="f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d" Oct 30 08:07:10.191861 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:10.191834 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d\": container with ID starting with f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d not found: ID does not exist" containerID="f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d" Oct 30 08:07:10.191994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.191868 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d"} err="failed to get container status \"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d\": rpc error: code = NotFound desc = could not find container \"f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d\": container with ID starting with f4af63a51729d46a00db5d8105f4f795cf5aab63803a17d0f140bbd4a725025d not found: ID does not exist" Oct 30 08:07:10.238002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.237913 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kst\" (UniqueName: \"kubernetes.io/projected/007e648c-d5b5-4b6c-9ece-748e0b79cfd0-kube-api-access-n8kst\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:10.303234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.303200 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 08:07:10.315424 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.315386 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6051b2-5c9975d764-tdbbt"] Oct 30 08:07:10.383647 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.383605 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 08:07:10.387210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.387187 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6051b2-5f7459db55-wz7dd"] Oct 30 08:07:10.433024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.432987 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" path="/var/lib/kubelet/pods/007e648c-d5b5-4b6c-9ece-748e0b79cfd0/volumes" Oct 30 08:07:10.433448 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.433434 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" path="/var/lib/kubelet/pods/c853d65c-6510-4fc7-addd-a2057dddb2bd/volumes" Oct 30 08:07:10.910080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.910040 5894 patch_prober.go:29] interesting pod/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:07:10.910253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:10.910108 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:07:11.245868 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.245824 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:11.357024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.356980 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:11.378406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.378341 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:07:11.438479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.438362 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:11.455874 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.455835 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\") pod \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " Oct 30 08:07:11.456065 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.455927 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\" (UniqueName: \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\") pod \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " Oct 30 08:07:11.456065 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.455964 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\") pod \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\" (UID: \"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60\") " Oct 30 08:07:11.456944 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.456910 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca" pod "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" (UID: "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:11.492905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.492831 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm" pod "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" (UID: "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60"). InnerVolumeSpecName "kube-api-access-z2bqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:11.504390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.499772 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca" pod "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" (UID: "24cc4c3c-6553-4c40-9c5e-72c9d04d7c60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:11.564722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.561939 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:11.564722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.561977 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2bqm\" (UniqueName: \"kubernetes.io/projected/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-kube-api-access-z2bqm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:11.564722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:11.561992 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.183861 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.183824 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" event={"ID":"24cc4c3c-6553-4c40-9c5e-72c9d04d7c60","Type":"ContainerDied","Data":"67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659"} Oct 30 08:07:12.184050 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.183874 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67d626adf326616eaad216ea33290eee94085a4ba22aa3a3718fc59ad5ab2659" Oct 30 08:07:12.184050 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.183840 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk" Oct 30 08:07:12.716403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.716331 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 08:07:12.775290 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775255 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775334 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775373 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775418 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775475 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775571 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:12.775763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775595 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775629 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775763 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775721 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775777 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\") pod \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\" (UID: \"3f0f3c81-6db3-48cb-9930-3f3af3ef896e\") " Oct 30 08:07:12.775998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.775814 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:12.776265 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.776244 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.776265 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.776267 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.776466 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.776289 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:12.776466 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.776375 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:12.776761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.776734 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:12.778295 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.778269 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:12.778705 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.778680 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "kube-api-access-bk4xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:12.778909 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.778883 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:12.779586 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.779564 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo" pod "3f0f3c81-6db3-48cb-9930-3f3af3ef896e" (UID: "3f0f3c81-6db3-48cb-9930-3f3af3ef896e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:07:12.877272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877237 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877269 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877279 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk4xv\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-kube-api-access-bk4xv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877292 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877301 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877310 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:12.877556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:12.877318 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f0f3c81-6db3-48cb-9930-3f3af3ef896e-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:13.194068 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.194035 5894 generic.go:356] "Generic (PLEG): container finished" podID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerID="2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed" exitCode=0 Oct 30 08:07:13.194278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.194088 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" Oct 30 08:07:13.194278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.194121 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerDied","Data":"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed"} Oct 30 08:07:13.194278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.194181 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj" event={"ID":"3f0f3c81-6db3-48cb-9930-3f3af3ef896e","Type":"ContainerDied","Data":"3b854ea6149f9bb12be78061dacc2660837ddb2da90c66b121efcdb2c60bd935"} Oct 30 08:07:13.194278 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.194204 5894 scope.go:122] "RemoveContainer" containerID="2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed" Oct 30 08:07:13.225973 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.219346 5894 scope.go:122] "RemoveContainer" containerID="5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9" Oct 30 08:07:13.225973 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.221659 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:13.239919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.239897 5894 scope.go:122] "RemoveContainer" containerID="2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed" Oct 30 08:07:13.240228 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:13.240212 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed\": container with ID starting with 2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed not found: ID does not exist" containerID="2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed" Oct 30 08:07:13.240284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.240238 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed"} err="failed to get container status \"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed\": rpc error: code = NotFound desc = could not find container \"2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed\": container with ID starting with 2213b2a7e130ea832bf7c23256032f63815b9760c5f2cc91a36d0de23c48baed not found: ID does not exist" Oct 30 08:07:13.240284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.240255 5894 scope.go:122] "RemoveContainer" containerID="5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9" Oct 30 08:07:13.240442 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:13.240428 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9\": container with ID starting with 5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9 not found: ID does not exist" containerID="5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9" Oct 30 08:07:13.240496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.240445 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9"} err="failed to get container status \"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9\": rpc error: code = NotFound desc = could not find container \"5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9\": container with ID starting with 5b9faa6b751a818761de2bd1c316cc853a44265e99bf3ca0b9f8e5235ecd98a9 not found: ID does not exist" Oct 30 08:07:13.283513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.283478 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 08:07:13.292427 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.292390 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6051b2-openshift-default-7d4db6d9d5-g4vdj"] Oct 30 08:07:13.786686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.786620 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr"] Oct 30 08:07:13.787276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787258 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" Oct 30 08:07:13.787320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787276 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" Oct 30 08:07:13.787320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787290 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:13.787320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787295 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:13.787320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787308 5894 cpu_manager.go:414] "containerMap: removing container" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" Oct 30 08:07:13.787320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787314 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787329 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787334 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787343 5894 cpu_manager.go:414] "containerMap: removing container" podUID="24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" containerName="revoke-keys" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787347 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" containerName="revoke-keys" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787434 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="007e648c-d5b5-4b6c-9ece-748e0b79cfd0" containerName="payload-pre-processing" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787443 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787451 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" containerName="revoke-keys" Oct 30 08:07:13.787502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.787464 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c853d65c-6510-4fc7-addd-a2057dddb2bd" containerName="payload-processing" Oct 30 08:07:13.792528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.792508 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.836388 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.836358 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:07:13.836620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.836512 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr"] Oct 30 08:07:13.836620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.836537 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:07:13.836620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.836584 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:07:13.836620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.836612 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:07:13.890485 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.890446 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.890736 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.890512 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddrn\" (UniqueName: \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.890892 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.890757 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.992332 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.992286 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.992553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.992428 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.992553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.992500 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddrn\" (UniqueName: \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.993027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.993007 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:13.995012 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:13.994991 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:14.019577 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:14.019541 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddrn\" (UniqueName: \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\") pod \"maas-api-revoke-sub-6ea667b3-r82zr\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:14.106889 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:14.106805 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:14.418555 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:14.418524 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e52df3d_9eb3_41f5_acdb_42670fe8791e.slice/crio-1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6 WatchSource:0}: Error finding container 1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6: Status 404 returned error can't find the container with id 1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6 Oct 30 08:07:14.419153 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:14.418955 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr"] Oct 30 08:07:14.432564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:14.432532 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" path="/var/lib/kubelet/pods/3f0f3c81-6db3-48cb-9930-3f3af3ef896e/volumes" Oct 30 08:07:15.213999 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:15.213968 5894 generic.go:356] "Generic (PLEG): container finished" podID="9e52df3d-9eb3-41f5-acdb-42670fe8791e" containerID="5ab4a51f5d0c3f055beec5ccb4b9800d74d8b31cc7dc161e8be8ae338412cd47" exitCode=0 Oct 30 08:07:15.214406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:15.214047 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" event={"ID":"9e52df3d-9eb3-41f5-acdb-42670fe8791e","Type":"ContainerDied","Data":"5ab4a51f5d0c3f055beec5ccb4b9800d74d8b31cc7dc161e8be8ae338412cd47"} Oct 30 08:07:15.214406 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:15.214098 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" event={"ID":"9e52df3d-9eb3-41f5-acdb-42670fe8791e","Type":"ContainerStarted","Data":"1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6"} Oct 30 08:07:15.230235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:15.230207 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:16.382806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.382779 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:16.523034 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.522956 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\" (UniqueName: \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\") pod \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " Oct 30 08:07:16.523034 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.523005 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\") pod \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " Oct 30 08:07:16.523282 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.523072 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\") pod \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\" (UID: \"9e52df3d-9eb3-41f5-acdb-42670fe8791e\") " Oct 30 08:07:16.523418 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.523381 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca" pod "9e52df3d-9eb3-41f5-acdb-42670fe8791e" (UID: "9e52df3d-9eb3-41f5-acdb-42670fe8791e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:16.523809 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.523787 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:16.525154 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.525128 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn" pod "9e52df3d-9eb3-41f5-acdb-42670fe8791e" (UID: "9e52df3d-9eb3-41f5-acdb-42670fe8791e"). InnerVolumeSpecName "kube-api-access-wddrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:16.525234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.525199 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca" pod "9e52df3d-9eb3-41f5-acdb-42670fe8791e" (UID: "9e52df3d-9eb3-41f5-acdb-42670fe8791e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:16.625073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.625032 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wddrn\" (UniqueName: \"kubernetes.io/projected/9e52df3d-9eb3-41f5-acdb-42670fe8791e-kube-api-access-wddrn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:16.625073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:16.625066 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e52df3d-9eb3-41f5-acdb-42670fe8791e-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:17.169646 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:17.169596 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:17.239950 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:17.239917 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" event={"ID":"9e52df3d-9eb3-41f5-acdb-42670fe8791e","Type":"ContainerDied","Data":"1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6"} Oct 30 08:07:17.240139 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:17.239995 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db336ad6b626615ba25afefe8b4b01e53aa8b9063e6ffaa2a793318445aa5a6" Oct 30 08:07:17.240139 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:17.239929 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ea667b3-r82zr" Oct 30 08:07:18.608541 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.608500 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:18.938199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.938124 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf"] Oct 30 08:07:18.938788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.938770 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9e52df3d-9eb3-41f5-acdb-42670fe8791e" containerName="revoke-keys" Oct 30 08:07:18.938847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.938789 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e52df3d-9eb3-41f5-acdb-42670fe8791e" containerName="revoke-keys" Oct 30 08:07:18.938906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.938896 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0f3c81-6db3-48cb-9930-3f3af3ef896e" containerName="istio-proxy" Oct 30 08:07:18.938941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.938914 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e52df3d-9eb3-41f5-acdb-42670fe8791e" containerName="revoke-keys" Oct 30 08:07:18.943024 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.942999 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:18.961318 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.961284 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:07:18.963854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.963823 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf"] Oct 30 08:07:18.976245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.975361 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:07:18.976245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.975500 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:07:18.976245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:18.976211 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:07:19.069416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.069374 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.069602 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.069523 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcpl\" (UniqueName: \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.069724 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.069620 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.171206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.170968 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.171206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.171141 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcpl\" (UniqueName: \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.171206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.171186 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.171744 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.171719 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.173767 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.173738 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.196888 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.196797 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcpl\" (UniqueName: \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\") pod \"maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.257227 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.257188 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:19.293492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.288541 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:19.608814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.608782 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf"] Oct 30 08:07:19.609468 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:19.609440 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4f1934_c46f_4269_8ea3_4b222239ee7c.slice/crio-2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f WatchSource:0}: Error finding container 2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f: Status 404 returned error can't find the container with id 2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f Oct 30 08:07:19.694114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.694077 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 08:07:19.730883 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.730846 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 08:07:19.866855 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:19.866685 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:20.085735 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:20.085698 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:20.265144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:20.265054 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" event={"ID":"6a4f1934-c46f-4269-8ea3-4b222239ee7c","Type":"ContainerStarted","Data":"c02b64118e349e80b2adfce2c9f0c1e4d2bc9eea1175e1f871df0ee0aff00f70"} Oct 30 08:07:20.265304 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:20.265147 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" event={"ID":"6a4f1934-c46f-4269-8ea3-4b222239ee7c","Type":"ContainerStarted","Data":"2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f"} Oct 30 08:07:20.314387 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:20.314310 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" podStartSLOduration=2.314290402 podStartE2EDuration="2.314290402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:20.310500852 +0000 UTC m=+4204.001418290" watchObservedRunningTime="2026-10-30 08:07:20.314290402 +0000 UTC m=+4204.005207840" Oct 30 08:07:21.274655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:21.274597 5894 generic.go:356] "Generic (PLEG): container finished" podID="6a4f1934-c46f-4269-8ea3-4b222239ee7c" containerID="c02b64118e349e80b2adfce2c9f0c1e4d2bc9eea1175e1f871df0ee0aff00f70" exitCode=0 Oct 30 08:07:21.275083 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:21.274695 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" event={"ID":"6a4f1934-c46f-4269-8ea3-4b222239ee7c","Type":"ContainerDied","Data":"c02b64118e349e80b2adfce2c9f0c1e4d2bc9eea1175e1f871df0ee0aff00f70"} Oct 30 08:07:21.335132 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:21.335091 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:21.929210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:21.929171 5894 scope.go:122] "RemoveContainer" containerID="9f9693bd1ab56bed9fec0b50687b7175d07edc715b701255c93fc5fab7139f06" Oct 30 08:07:22.287779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.287748 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:07:22.444934 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.444908 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:22.619235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.619127 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\") pod \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " Oct 30 08:07:22.619518 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.619491 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\") pod \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " Oct 30 08:07:22.619598 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.619559 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\" (UniqueName: \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\") pod \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\" (UID: \"6a4f1934-c46f-4269-8ea3-4b222239ee7c\") " Oct 30 08:07:22.622568 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.622527 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl" pod "6a4f1934-c46f-4269-8ea3-4b222239ee7c" (UID: "6a4f1934-c46f-4269-8ea3-4b222239ee7c"). InnerVolumeSpecName "kube-api-access-vtcpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:22.633668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.632899 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca" pod "6a4f1934-c46f-4269-8ea3-4b222239ee7c" (UID: "6a4f1934-c46f-4269-8ea3-4b222239ee7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:22.643854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.641133 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcpl\" (UniqueName: \"kubernetes.io/projected/6a4f1934-c46f-4269-8ea3-4b222239ee7c-kube-api-access-vtcpl\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:22.643854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.641168 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:22.660292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.660128 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca" pod "6a4f1934-c46f-4269-8ea3-4b222239ee7c" (UID: "6a4f1934-c46f-4269-8ea3-4b222239ee7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:22.742722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:22.742676 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a4f1934-c46f-4269-8ea3-4b222239ee7c-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:23.293085 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:23.293052 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" Oct 30 08:07:23.293520 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:23.293058 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf" event={"ID":"6a4f1934-c46f-4269-8ea3-4b222239ee7c","Type":"ContainerDied","Data":"2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f"} Oct 30 08:07:23.293520 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:23.293191 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8938608e465d388071c2f09a552fe409882dc7587f655394f99e94fde8919f" Oct 30 08:07:23.511218 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:23.511183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:24.694587 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:24.694545 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" containerID="cri-o://65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca" gracePeriod=25 Oct 30 08:07:24.731134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:24.731085 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" containerID="cri-o://7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36" gracePeriod=25 Oct 30 08:07:25.553942 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:25.553906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:26.401357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.401317 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" podUID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" containerName="main" containerID="cri-o://257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50" gracePeriod=45 Oct 30 08:07:26.490583 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.490543 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:26.490828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.490543 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:26.667775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.667710 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:07:26.701258 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701216 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701307 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701342 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\" (UniqueName: \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701382 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701456 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\" (UniqueName: \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701544 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:26.701629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701570 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:26.701629 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701590 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\") pod \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\" (UID: \"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07\") " Oct 30 08:07:26.701852 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.701792 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:26.702048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.702029 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:26.702048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.702048 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:26.702210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.702057 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:26.703802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.703773 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "kube-api-access-d9csh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:26.703917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.703795 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:26.703917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.703805 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs" pod "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" (UID: "3093fbfe-5c48-49a7-b8e8-78bee4a6cc07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:26.803215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.803174 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:26.803215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.803206 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:26.803215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:26.803218 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9csh\" (UniqueName: \"kubernetes.io/projected/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07-kube-api-access-d9csh\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:27.332022 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.331987 5894 generic.go:356] "Generic (PLEG): container finished" podID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" containerID="257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50" exitCode=0 Oct 30 08:07:27.332236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.332037 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" Oct 30 08:07:27.332236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.332053 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" event={"ID":"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07","Type":"ContainerDied","Data":"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50"} Oct 30 08:07:27.332236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.332097 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9" event={"ID":"3093fbfe-5c48-49a7-b8e8-78bee4a6cc07","Type":"ContainerDied","Data":"54d841099987873a547fe92dc5c299e13e17338b254290e4f0fb8a19debd4cbd"} Oct 30 08:07:27.332236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.332117 5894 scope.go:122] "RemoveContainer" containerID="257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50" Oct 30 08:07:27.344919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.344884 5894 scope.go:122] "RemoveContainer" containerID="257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50" Oct 30 08:07:27.345337 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:27.345314 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50\": container with ID starting with 257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50 not found: ID does not exist" containerID="257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50" Oct 30 08:07:27.345421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.345346 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50"} err="failed to get container status \"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50\": rpc error: code = NotFound desc = could not find container \"257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50\": container with ID starting with 257ec60085d05d6fdb87cd942886065ed2be23632f91759d9ce0b88ad5f38e50 not found: ID does not exist" Oct 30 08:07:27.396426 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.396388 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:07:27.400463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.400436 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-2e8b2a/rate-test-model-2e8b2a-kserve-6f75c4f9b5-pnlx9"] Oct 30 08:07:27.411814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.411780 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:27.668180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.668084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:27.829878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:27.829839 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:28.431901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:28.431865 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" path="/var/lib/kubelet/pods/3093fbfe-5c48-49a7-b8e8-78bee4a6cc07/volumes" Oct 30 08:07:29.342254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:29.342215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:29.599405 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:29.599320 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:29.717204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:29.717163 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:30.959589 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:30.959550 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:31.540923 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:31.540890 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:31.666762 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:31.666720 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:31.690500 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:31.690462 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf"] Oct 30 08:07:31.707478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:31.707430 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1e0880-55tnf"] Oct 30 08:07:31.801537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:31.801432 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:32.182960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:32.182873 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:32.431880 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:32.431847 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a4f1934-c46f-4269-8ea3-4b222239ee7c" path="/var/lib/kubelet/pods/6a4f1934-c46f-4269-8ea3-4b222239ee7c/volumes" Oct 30 08:07:33.506600 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:33.506559 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:34.003756 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.003719 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:34.158930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.158899 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:34.240916 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.240871 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk"] Oct 30 08:07:34.255444 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.255359 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-bb2e8f-zkqqk"] Oct 30 08:07:34.278565 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.278530 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:34.431931 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.431898 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24cc4c3c-6553-4c40-9c5e-72c9d04d7c60" path="/var/lib/kubelet/pods/24cc4c3c-6553-4c40-9c5e-72c9d04d7c60/volumes" Oct 30 08:07:34.679135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.679100 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 08:07:34.753687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.753654 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:34.865024 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:34.864846 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:07:34.916101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:34.916064 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:36.491086 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:36.491033 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:36.491548 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:36.491098 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:36.526612 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:36.526567 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:38.339340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:38.339298 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:38.707073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:38.706994 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:38.818892 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:38.818849 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:39.049583 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.049547 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:39.538931 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.534624 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:39.630946 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.630914 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g"] Oct 30 08:07:39.631526 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631510 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" containerName="main" Oct 30 08:07:39.631588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631526 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" containerName="main" Oct 30 08:07:39.631588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631539 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6a4f1934-c46f-4269-8ea3-4b222239ee7c" containerName="revoke-keys" Oct 30 08:07:39.631588 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631544 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a4f1934-c46f-4269-8ea3-4b222239ee7c" containerName="revoke-keys" Oct 30 08:07:39.631737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631671 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3093fbfe-5c48-49a7-b8e8-78bee4a6cc07" containerName="main" Oct 30 08:07:39.631737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.631682 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a4f1934-c46f-4269-8ea3-4b222239ee7c" containerName="revoke-keys" Oct 30 08:07:39.637018 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.636992 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.655144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.655113 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g"] Oct 30 08:07:39.729276 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.729232 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxn8\" (UniqueName: \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.729497 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.729291 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.729497 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.729475 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.745130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.745099 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:07:39.745370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.745351 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:07:39.745805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.745782 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:07:39.747138 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.747119 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:07:39.803086 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.803007 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442"] Oct 30 08:07:39.807569 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.807527 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.828502 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.828472 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442"] Oct 30 08:07:39.831075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831040 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-264lp\" (UniqueName: \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.831292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831272 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.831475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831442 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.831616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831531 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxn8\" (UniqueName: \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.831616 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831592 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.831802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.831683 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.832543 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.832518 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.835591 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.835567 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.859954 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.859909 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxn8\" (UniqueName: \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\") pod \"maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.933339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.933303 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.933549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.933443 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-264lp\" (UniqueName: \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.933549 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.933472 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.934189 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.934156 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.935881 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.935857 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:39.955980 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.955947 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:39.961270 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:39.961242 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-264lp\" (UniqueName: \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\") pod \"maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:40.122298 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.122215 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:40.288703 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:40.288665 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2b1b56_30c8_47ad_ae4e_076a3f9f37ad.slice/crio-b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037 WatchSource:0}: Error finding container b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037: Status 404 returned error can't find the container with id b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037 Oct 30 08:07:40.289689 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.289655 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g"] Oct 30 08:07:40.434249 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:40.434213 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf23c8a_12f3_498c_b127_dddeeeec84af.slice/crio-b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7 WatchSource:0}: Error finding container b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7: Status 404 returned error can't find the container with id b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7 Oct 30 08:07:40.434403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.434261 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442"] Oct 30 08:07:40.434991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.434957 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:40.502340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.502306 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" event={"ID":"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad","Type":"ContainerStarted","Data":"b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037"} Oct 30 08:07:40.503826 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:40.503790 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" event={"ID":"edf23c8a-12f3-498c-b127-dddeeeec84af","Type":"ContainerStarted","Data":"b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7"} Oct 30 08:07:41.513737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:41.513699 5894 generic.go:356] "Generic (PLEG): container finished" podID="ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" containerID="022b42a47b1235295273b1f54b8e056f9ce8e59642fed88aa7d77f08733e9432" exitCode=0 Oct 30 08:07:41.514234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:41.513751 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" event={"ID":"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad","Type":"ContainerDied","Data":"022b42a47b1235295273b1f54b8e056f9ce8e59642fed88aa7d77f08733e9432"} Oct 30 08:07:41.515579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:41.515554 5894 generic.go:356] "Generic (PLEG): container finished" podID="edf23c8a-12f3-498c-b127-dddeeeec84af" containerID="1b41e335267dea3e255005b3070fcbd1bbcc64df9e3108cd5bd965d078a42f37" exitCode=0 Oct 30 08:07:41.515726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:41.515593 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" event={"ID":"edf23c8a-12f3-498c-b127-dddeeeec84af","Type":"ContainerDied","Data":"1b41e335267dea3e255005b3070fcbd1bbcc64df9e3108cd5bd965d078a42f37"} Oct 30 08:07:42.053573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.053538 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:42.173119 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.173084 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:42.460477 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.460394 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:42.603421 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.603305 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:42.757698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.757668 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:42.793367 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.793333 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\" (UniqueName: \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\") pod \"edf23c8a-12f3-498c-b127-dddeeeec84af\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " Oct 30 08:07:42.793561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.793425 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\") pod \"edf23c8a-12f3-498c-b127-dddeeeec84af\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " Oct 30 08:07:42.793671 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.793617 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\") pod \"edf23c8a-12f3-498c-b127-dddeeeec84af\" (UID: \"edf23c8a-12f3-498c-b127-dddeeeec84af\") " Oct 30 08:07:42.793831 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.793802 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca" pod "edf23c8a-12f3-498c-b127-dddeeeec84af" (UID: "edf23c8a-12f3-498c-b127-dddeeeec84af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:42.794117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.794095 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:42.795696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.795664 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca" pod "edf23c8a-12f3-498c-b127-dddeeeec84af" (UID: "edf23c8a-12f3-498c-b127-dddeeeec84af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:42.795810 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.795690 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp" pod "edf23c8a-12f3-498c-b127-dddeeeec84af" (UID: "edf23c8a-12f3-498c-b127-dddeeeec84af"). InnerVolumeSpecName "kube-api-access-264lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:42.820526 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.820494 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:42.895462 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.895419 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\") pod \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " Oct 30 08:07:42.895694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.895506 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\") pod \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " Oct 30 08:07:42.895771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.895721 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\" (UniqueName: \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\") pod \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\" (UID: \"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad\") " Oct 30 08:07:42.896002 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.895968 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca" pod "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" (UID: "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:42.896349 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.896326 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edf23c8a-12f3-498c-b127-dddeeeec84af-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:42.896467 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.896354 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-264lp\" (UniqueName: \"kubernetes.io/projected/edf23c8a-12f3-498c-b127-dddeeeec84af-kube-api-access-264lp\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:42.896467 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.896371 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:42.898539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.898390 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8" pod "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" (UID: "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad"). InnerVolumeSpecName "kube-api-access-srxn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:42.898539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.898482 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca" pod "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" (UID: "ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:42.997293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.997202 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srxn8\" (UniqueName: \"kubernetes.io/projected/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-kube-api-access-srxn8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:42.997293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:42.997235 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:43.068459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.068421 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf"] Oct 30 08:07:43.069089 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069073 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" containerName="revoke-keys" Oct 30 08:07:43.069156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069090 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" containerName="revoke-keys" Oct 30 08:07:43.069156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069111 5894 cpu_manager.go:414] "containerMap: removing container" podUID="edf23c8a-12f3-498c-b127-dddeeeec84af" containerName="revoke-keys" Oct 30 08:07:43.069156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069116 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edf23c8a-12f3-498c-b127-dddeeeec84af" containerName="revoke-keys" Oct 30 08:07:43.069254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069213 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="edf23c8a-12f3-498c-b127-dddeeeec84af" containerName="revoke-keys" Oct 30 08:07:43.069254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.069235 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" containerName="revoke-keys" Oct 30 08:07:43.073188 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.073152 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.093020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.092985 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf"] Oct 30 08:07:43.105372 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.098510 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdmch\" (UniqueName: \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.105372 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.098623 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.105372 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.098845 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.200377 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.200342 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.200583 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.200476 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.200583 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.200536 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdmch\" (UniqueName: \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.201178 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.201156 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.202741 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.202722 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.226350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.226321 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdmch\" (UniqueName: \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.394683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.394590 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:43.540324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.540282 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" Oct 30 08:07:43.540542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.540282 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g" event={"ID":"ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad","Type":"ContainerDied","Data":"b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037"} Oct 30 08:07:43.540542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.540440 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b14f77446cdbf597cf4def2939e4ee7b46231c60fdef322f712eaf0a6b162037" Oct 30 08:07:43.541951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.541923 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" event={"ID":"edf23c8a-12f3-498c-b127-dddeeeec84af","Type":"ContainerDied","Data":"b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7"} Oct 30 08:07:43.541951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.541940 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442" Oct 30 08:07:43.542134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.541971 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b68534c48d9a3e872ec62e0de4b977680c5f990081c290b9409d7e12fbcddbd7" Oct 30 08:07:43.704960 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:07:43.704926 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc542d010_4cb5_467c_aaff_2e29cb36a3fe.slice/crio-15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f WatchSource:0}: Error finding container 15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f: Status 404 returned error can't find the container with id 15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f Oct 30 08:07:43.705956 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:43.705920 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf"] Oct 30 08:07:44.517080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:44.512418 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:44.551718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:44.551678 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" event={"ID":"c542d010-4cb5-467c-aaff-2e29cb36a3fe","Type":"ContainerStarted","Data":"8e696a7452ac2d63cb7cb5afc62e3328595a47b94f9d9cbc52e32d68535dfee1"} Oct 30 08:07:44.551909 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:44.551744 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" event={"ID":"c542d010-4cb5-467c-aaff-2e29cb36a3fe","Type":"ContainerStarted","Data":"15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f"} Oct 30 08:07:44.599217 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:44.599161 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" podStartSLOduration=1.599145553 podStartE2EDuration="1.599145553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:07:44.596659694 +0000 UTC m=+4228.287577127" watchObservedRunningTime="2026-10-30 08:07:44.599145553 +0000 UTC m=+4228.290062989" Oct 30 08:07:45.012586 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.012558 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 08:07:45.034866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.034839 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\" (UniqueName: \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\") pod \"59a864dd-601f-426a-beca-2a039bb0e46b\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " Oct 30 08:07:45.035013 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.034901 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\") pod \"59a864dd-601f-426a-beca-2a039bb0e46b\" (UID: \"59a864dd-601f-426a-beca-2a039bb0e46b\") " Oct 30 08:07:45.035323 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.035298 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume" pod "59a864dd-601f-426a-beca-2a039bb0e46b" (UID: "59a864dd-601f-426a-beca-2a039bb0e46b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:45.035484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.035469 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59a864dd-601f-426a-beca-2a039bb0e46b-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:45.037151 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.037127 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz" pod "59a864dd-601f-426a-beca-2a039bb0e46b" (UID: "59a864dd-601f-426a-beca-2a039bb0e46b"). InnerVolumeSpecName "kube-api-access-fwrdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:45.037269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.037152 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 08:07:45.136562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.136471 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\" (UniqueName: \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\") pod \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " Oct 30 08:07:45.136743 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.136615 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\") pod \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\" (UID: \"6895f08d-18b4-40ed-a137-891eb9f6dfb6\") " Oct 30 08:07:45.137023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.136972 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume" pod "6895f08d-18b4-40ed-a137-891eb9f6dfb6" (UID: "6895f08d-18b4-40ed-a137-891eb9f6dfb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:45.137226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.137205 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwrdz\" (UniqueName: \"kubernetes.io/projected/59a864dd-601f-426a-beca-2a039bb0e46b-kube-api-access-fwrdz\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:45.137288 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.137229 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6895f08d-18b4-40ed-a137-891eb9f6dfb6-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:45.139311 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.139285 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr" pod "6895f08d-18b4-40ed-a137-891eb9f6dfb6" (UID: "6895f08d-18b4-40ed-a137-891eb9f6dfb6"). InnerVolumeSpecName "kube-api-access-g6cqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:45.238575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.238536 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6cqr\" (UniqueName: \"kubernetes.io/projected/6895f08d-18b4-40ed-a137-891eb9f6dfb6-kube-api-access-g6cqr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:45.561366 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.561334 5894 generic.go:356] "Generic (PLEG): container finished" podID="c542d010-4cb5-467c-aaff-2e29cb36a3fe" containerID="8e696a7452ac2d63cb7cb5afc62e3328595a47b94f9d9cbc52e32d68535dfee1" exitCode=0 Oct 30 08:07:45.561539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.561424 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" event={"ID":"c542d010-4cb5-467c-aaff-2e29cb36a3fe","Type":"ContainerDied","Data":"8e696a7452ac2d63cb7cb5afc62e3328595a47b94f9d9cbc52e32d68535dfee1"} Oct 30 08:07:45.562779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.562755 5894 generic.go:356] "Generic (PLEG): container finished" podID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerID="7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36" exitCode=0 Oct 30 08:07:45.562919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.562784 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" event={"ID":"6895f08d-18b4-40ed-a137-891eb9f6dfb6","Type":"ContainerDied","Data":"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36"} Oct 30 08:07:45.562919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.562799 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" Oct 30 08:07:45.562919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.562829 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn" event={"ID":"6895f08d-18b4-40ed-a137-891eb9f6dfb6","Type":"ContainerDied","Data":"908c263e5d0dda0634a03eed953b5c4114a5ca1162e41b6ba7e72599cae0a4c9"} Oct 30 08:07:45.562919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.562851 5894 scope.go:122] "RemoveContainer" containerID="7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36" Oct 30 08:07:45.564314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.564294 5894 generic.go:356] "Generic (PLEG): container finished" podID="59a864dd-601f-426a-beca-2a039bb0e46b" containerID="65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca" exitCode=0 Oct 30 08:07:45.564429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.564326 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" event={"ID":"59a864dd-601f-426a-beca-2a039bb0e46b","Type":"ContainerDied","Data":"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca"} Oct 30 08:07:45.564429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.564363 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" Oct 30 08:07:45.564429 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.564376 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9" event={"ID":"59a864dd-601f-426a-beca-2a039bb0e46b","Type":"ContainerDied","Data":"66c45437318ff87e15f17d7142c5e019fe26db69d066416bb0dbc534f0b14a25"} Oct 30 08:07:45.574092 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.574072 5894 scope.go:122] "RemoveContainer" containerID="7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36" Oct 30 08:07:45.574421 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:45.574399 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36\": container with ID starting with 7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36 not found: ID does not exist" containerID="7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36" Oct 30 08:07:45.574500 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.574431 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36"} err="failed to get container status \"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36\": rpc error: code = NotFound desc = could not find container \"7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36\": container with ID starting with 7862af9e04172ef42173b68bf5777ff25b52b7c1d2e5a456398e6e2d84a61d36 not found: ID does not exist" Oct 30 08:07:45.574500 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.574457 5894 scope.go:122] "RemoveContainer" containerID="65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca" Oct 30 08:07:45.585652 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.585616 5894 scope.go:122] "RemoveContainer" containerID="65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca" Oct 30 08:07:45.585960 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:45.585941 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca\": container with ID starting with 65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca not found: ID does not exist" containerID="65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca" Oct 30 08:07:45.586031 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.585967 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca"} err="failed to get container status \"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca\": rpc error: code = NotFound desc = could not find container \"65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca\": container with ID starting with 65d599627284021c57d7684e0ff5d3ad7c4ef698b03e6182808a128568c66bca not found: ID does not exist" Oct 30 08:07:45.651485 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.651443 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 08:07:45.656354 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.656326 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bb2e8f-7cbf875c6d-zqtv9"] Oct 30 08:07:45.695606 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.695568 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 08:07:45.699006 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:45.698982 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-bb2e8f-5d5c7867cf-mmwqn"] Oct 30 08:07:46.247932 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.247894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:46.431925 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.431876 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" path="/var/lib/kubelet/pods/59a864dd-601f-426a-beca-2a039bb0e46b/volumes" Oct 30 08:07:46.432371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.432348 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" path="/var/lib/kubelet/pods/6895f08d-18b4-40ed-a137-891eb9f6dfb6/volumes" Oct 30 08:07:46.737471 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.737442 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:46.860434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.860342 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\") pod \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " Oct 30 08:07:46.860619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.860504 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\") pod \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " Oct 30 08:07:46.860619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.860563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\" (UniqueName: \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\") pod \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\" (UID: \"c542d010-4cb5-467c-aaff-2e29cb36a3fe\") " Oct 30 08:07:46.860806 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.860729 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca" pod "c542d010-4cb5-467c-aaff-2e29cb36a3fe" (UID: "c542d010-4cb5-467c-aaff-2e29cb36a3fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:07:46.861118 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.861094 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:46.862771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.862749 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca" pod "c542d010-4cb5-467c-aaff-2e29cb36a3fe" (UID: "c542d010-4cb5-467c-aaff-2e29cb36a3fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:46.862850 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.862779 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch" pod "c542d010-4cb5-467c-aaff-2e29cb36a3fe" (UID: "c542d010-4cb5-467c-aaff-2e29cb36a3fe"). InnerVolumeSpecName "kube-api-access-wdmch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:46.962327 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.962279 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c542d010-4cb5-467c-aaff-2e29cb36a3fe-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:46.962327 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:46.962318 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdmch\" (UniqueName: \"kubernetes.io/projected/c542d010-4cb5-467c-aaff-2e29cb36a3fe-kube-api-access-wdmch\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:47.583220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:47.583188 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" Oct 30 08:07:47.583220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:47.583199 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf" event={"ID":"c542d010-4cb5-467c-aaff-2e29cb36a3fe","Type":"ContainerDied","Data":"15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f"} Oct 30 08:07:47.583735 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:47.583251 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15afb1d60009fa391c7b557632e56623485de3b0fcad624242daf0ac59eb9d6f" Oct 30 08:07:48.140840 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:48.140804 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:48.256936 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:48.256906 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:49.703324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:49.703273 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" podUID="ffaf9e24-8a22-4cf5-8595-40e092726864" containerName="main" containerID="cri-o://58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101" gracePeriod=45 Oct 30 08:07:49.993093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:49.993066 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 08:07:50.046266 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.046224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:50.091497 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.091431 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 08:07:50.101262 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101218 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\" (UniqueName: \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.101454 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101310 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.101454 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101381 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.101454 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101422 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.101682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101483 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.101682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.101552 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\") pod \"ffaf9e24-8a22-4cf5-8595-40e092726864\" (UID: \"ffaf9e24-8a22-4cf5-8595-40e092726864\") " Oct 30 08:07:50.102420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.102386 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:50.102597 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.102577 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:50.103696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.103666 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:50.105009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.104980 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "kube-api-access-774k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:07:50.106603 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.105711 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:07:50.106603 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.106567 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs" pod "ffaf9e24-8a22-4cf5-8595-40e092726864" (UID: "ffaf9e24-8a22-4cf5-8595-40e092726864"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:07:50.107473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.107450 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 08:07:50.203697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203657 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-774k9\" (UniqueName: \"kubernetes.io/projected/ffaf9e24-8a22-4cf5-8595-40e092726864-kube-api-access-774k9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.203697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203698 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.203906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203718 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffaf9e24-8a22-4cf5-8595-40e092726864-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.203906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203731 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.203906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203746 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.203906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.203759 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffaf9e24-8a22-4cf5-8595-40e092726864-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:07:50.217775 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.217697 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:50.351106 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.351067 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:50.613970 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.613924 5894 generic.go:356] "Generic (PLEG): container finished" podID="ffaf9e24-8a22-4cf5-8595-40e092726864" containerID="58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101" exitCode=0 Oct 30 08:07:50.614164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.614062 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" event={"ID":"ffaf9e24-8a22-4cf5-8595-40e092726864","Type":"ContainerDied","Data":"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101"} Oct 30 08:07:50.614164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.614100 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" event={"ID":"ffaf9e24-8a22-4cf5-8595-40e092726864","Type":"ContainerDied","Data":"f8694fbb4383922268ef22e8c7f4f2fd31a4e8c168329b1ff9a1c3c6097f6bd0"} Oct 30 08:07:50.614164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.614121 5894 scope.go:122] "RemoveContainer" containerID="58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101" Oct 30 08:07:50.614340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.614254 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx" Oct 30 08:07:50.636251 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.636220 5894 scope.go:122] "RemoveContainer" containerID="58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101" Oct 30 08:07:50.636599 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:07:50.636576 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101\": container with ID starting with 58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101 not found: ID does not exist" containerID="58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101" Oct 30 08:07:50.636681 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.636610 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101"} err="failed to get container status \"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101\": rpc error: code = NotFound desc = could not find container \"58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101\": container with ID starting with 58f4c0e7ba1acd9c928b57074b2334affe15f2e588a42a901239c9ebeece1101 not found: ID does not exist" Oct 30 08:07:50.686273 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.686232 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 08:07:50.700446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.700400 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bf563a/sub-test-model-bf563a-kserve-fbf5dd867-799qx"] Oct 30 08:07:50.901893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.901806 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 08:07:50.933595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:50.933561 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 08:07:51.054046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:51.053947 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:51.178481 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:51.178400 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:52.042370 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:52.042334 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:52.431152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:52.431063 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffaf9e24-8a22-4cf5-8595-40e092726864" path="/var/lib/kubelet/pods/ffaf9e24-8a22-4cf5-8595-40e092726864/volumes" Oct 30 08:07:53.490239 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:53.490203 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 08:07:53.639492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:53.639452 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:53.770869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:53.770765 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:53.901792 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:53.901758 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:55.097343 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.097302 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" containerID="cri-o://97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38" gracePeriod=25 Oct 30 08:07:55.108403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.108355 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" containerID="cri-o://0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386" gracePeriod=25 Oct 30 08:07:55.398561 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.398471 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:55.398754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.398481 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:55.720685 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.720565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:55.902498 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.902453 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" containerID="cri-o://82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d" gracePeriod=25 Oct 30 08:07:55.934046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:55.934010 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" containerID="cri-o://a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781" gracePeriod=25 Oct 30 08:07:56.711302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:56.711261 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:56.711759 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:56.711263 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:07:57.541419 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:57.541373 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:07:58.181725 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:58.181688 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:07:58.491216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:58.491116 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" containerID="cri-o://3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd" gracePeriod=25 Oct 30 08:07:59.365642 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:07:59.365606 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:01.141034 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:01.140992 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:02.972731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:02.972689 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:03.342246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:03.342206 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:04.624938 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:04.624886 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:04.857476 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:04.857436 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:08:05.027252 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.027176 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:05.168717 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.168682 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:05.296518 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.296419 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:05.323080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.323047 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g"] Oct 30 08:08:05.337049 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.337012 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e193ee-pzl2g"] Oct 30 08:08:05.427962 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.427916 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:05.428199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.428163 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:05.710269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.710172 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:05.833245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.833210 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:05.870534 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.870500 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442"] Oct 30 08:08:05.873705 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.873673 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-b096ec-vq442"] Oct 30 08:08:05.951908 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:05.951876 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:06.367542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:06.367503 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:06.431614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:06.431579 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad" path="/var/lib/kubelet/pods/ae2b1b56-30c8-47ad-ae4e-076a3f9f37ad/volumes" Oct 30 08:08:06.432093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:06.432078 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edf23c8a-12f3-498c-b127-dddeeeec84af" path="/var/lib/kubelet/pods/edf23c8a-12f3-498c-b127-dddeeeec84af/volumes" Oct 30 08:08:06.711492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:06.711393 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:06.711492 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:06.711405 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:07.930052 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:07.930016 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:09.741036 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:09.740999 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:11.558428 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:11.558393 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:12.829779 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:12.829741 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:12.960085 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:12.960044 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:13.139145 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.139065 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 08:08:13.139329 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.139293 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" containerID="cri-o://55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb" gracePeriod=30 Oct 30 08:08:13.175909 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.175875 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:13.299227 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.299181 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:13.342302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.342263 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:13.736648 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.736607 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:13.860194 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.860156 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:13.881057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.881024 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf"] Oct 30 08:08:13.884577 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.884554 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-d89593-pkvzf"] Oct 30 08:08:13.976107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:13.976074 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:14.033029 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.032997 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p"] Oct 30 08:08:14.033573 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033560 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ffaf9e24-8a22-4cf5-8595-40e092726864" containerName="main" Oct 30 08:08:14.033624 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033575 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffaf9e24-8a22-4cf5-8595-40e092726864" containerName="main" Oct 30 08:08:14.033624 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033595 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" Oct 30 08:08:14.033624 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033600 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033629 5894 cpu_manager.go:414] "containerMap: removing container" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033648 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033658 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c542d010-4cb5-467c-aaff-2e29cb36a3fe" containerName="revoke-keys" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033663 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c542d010-4cb5-467c-aaff-2e29cb36a3fe" containerName="revoke-keys" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033747 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c542d010-4cb5-467c-aaff-2e29cb36a3fe" containerName="revoke-keys" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033758 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffaf9e24-8a22-4cf5-8595-40e092726864" containerName="main" Oct 30 08:08:14.033771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033767 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6895f08d-18b4-40ed-a137-891eb9f6dfb6" containerName="payload-pre-processing" Oct 30 08:08:14.033999 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.033778 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a864dd-601f-426a-beca-2a039bb0e46b" containerName="payload-processing" Oct 30 08:08:14.039226 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.039202 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.064176 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.064140 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:08:14.071480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.067628 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:08:14.071480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.068678 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:08:14.071480 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.069179 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:08:14.073884 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.073852 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p"] Oct 30 08:08:14.112488 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.112453 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:14.153329 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.153295 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgtt5\" (UniqueName: \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.153530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.153348 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.153646 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.153605 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.255181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.255148 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgtt5\" (UniqueName: \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.255375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.255198 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.255441 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.255382 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.255952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.255924 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.257749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.257725 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.289496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.289399 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgtt5\" (UniqueName: \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.362010 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.361971 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:14.378381 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.378348 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:14.460687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.454004 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c542d010-4cb5-467c-aaff-2e29cb36a3fe" path="/var/lib/kubelet/pods/c542d010-4cb5-467c-aaff-2e29cb36a3fe/volumes" Oct 30 08:08:14.494282 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.494250 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:14.709004 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:14.708960 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07b900c_937f_4553_a570_fe9842fd2abc.slice/crio-ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59 WatchSource:0}: Error finding container ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59: Status 404 returned error can't find the container with id ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59 Oct 30 08:08:14.710125 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.710098 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p"] Oct 30 08:08:14.815272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.815238 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" event={"ID":"b07b900c-937f-4553-a570-fe9842fd2abc","Type":"ContainerStarted","Data":"ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59"} Oct 30 08:08:14.946877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.946844 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr"] Oct 30 08:08:14.952371 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.952347 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:14.969501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:14.969473 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr"] Oct 30 08:08:15.095882 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.095846 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.096075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.095992 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mtt\" (UniqueName: \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.096075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.096035 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.144367 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.144319 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:15.197666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.197619 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.197872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.197736 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mtt\" (UniqueName: \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.197872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.197820 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.198465 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.198447 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.200057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.200040 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.224710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.224612 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mtt\" (UniqueName: \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\") pod \"maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.268870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.266491 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:15.268870 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.268064 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:15.419176 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.419149 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 08:08:15.422860 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.422832 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 08:08:15.501592 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.501513 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\") pod \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " Oct 30 08:08:15.501865 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.501626 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\" (UniqueName: \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\") pod \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\" (UID: \"3ffde2fa-572a-473d-95d7-c9cf252dfb42\") " Oct 30 08:08:15.502023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.501994 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume" pod "3ffde2fa-572a-473d-95d7-c9cf252dfb42" (UID: "3ffde2fa-572a-473d-95d7-c9cf252dfb42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:15.502218 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.502161 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ffde2fa-572a-473d-95d7-c9cf252dfb42-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:15.503974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.503944 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc" pod "3ffde2fa-572a-473d-95d7-c9cf252dfb42" (UID: "3ffde2fa-572a-473d-95d7-c9cf252dfb42"). InnerVolumeSpecName "kube-api-access-x75sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:15.603390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.603353 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\" (UniqueName: \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\") pod \"dce487fc-aaea-455b-9fe2-109bda57ec8b\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " Oct 30 08:08:15.603572 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.603424 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\") pod \"dce487fc-aaea-455b-9fe2-109bda57ec8b\" (UID: \"dce487fc-aaea-455b-9fe2-109bda57ec8b\") " Oct 30 08:08:15.603872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.603843 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume" pod "dce487fc-aaea-455b-9fe2-109bda57ec8b" (UID: "dce487fc-aaea-455b-9fe2-109bda57ec8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:15.604032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.604001 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dce487fc-aaea-455b-9fe2-109bda57ec8b-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:15.604032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.604019 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x75sc\" (UniqueName: \"kubernetes.io/projected/3ffde2fa-572a-473d-95d7-c9cf252dfb42-kube-api-access-x75sc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:15.606356 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.606037 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc" pod "dce487fc-aaea-455b-9fe2-109bda57ec8b" (UID: "dce487fc-aaea-455b-9fe2-109bda57ec8b"). InnerVolumeSpecName "kube-api-access-q5lmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:15.611138 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.611112 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr"] Oct 30 08:08:15.611259 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:15.611201 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06128f7f_8928_494f_9e76_0cf26538fb51.slice/crio-6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a WatchSource:0}: Error finding container 6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a: Status 404 returned error can't find the container with id 6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a Oct 30 08:08:15.623674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.623624 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:08:15.705284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.705247 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lmc\" (UniqueName: \"kubernetes.io/projected/dce487fc-aaea-455b-9fe2-109bda57ec8b-kube-api-access-q5lmc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:15.823558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.823516 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" event={"ID":"06128f7f-8928-494f-9e76-0cf26538fb51","Type":"ContainerStarted","Data":"6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a"} Oct 30 08:08:15.825107 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.825074 5894 generic.go:356] "Generic (PLEG): container finished" podID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerID="0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386" exitCode=0 Oct 30 08:08:15.825250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.825149 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" Oct 30 08:08:15.825250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.825152 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" event={"ID":"dce487fc-aaea-455b-9fe2-109bda57ec8b","Type":"ContainerDied","Data":"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386"} Oct 30 08:08:15.825397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.825262 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" event={"ID":"dce487fc-aaea-455b-9fe2-109bda57ec8b","Type":"ContainerDied","Data":"8d082ea8a32ac8dd4fa99ccf70639d24ff1c37bad6925db89acad5ec3dcfe5ef"} Oct 30 08:08:15.825397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.825291 5894 scope.go:122] "RemoveContainer" containerID="0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386" Oct 30 08:08:15.827093 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.827059 5894 generic.go:356] "Generic (PLEG): container finished" podID="b07b900c-937f-4553-a570-fe9842fd2abc" containerID="282ed586565094fb30cbc430f9111d614cb38ddd29bf1fc93f213ea64f6d5d64" exitCode=0 Oct 30 08:08:15.827220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.827142 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" event={"ID":"b07b900c-937f-4553-a570-fe9842fd2abc","Type":"ContainerDied","Data":"282ed586565094fb30cbc430f9111d614cb38ddd29bf1fc93f213ea64f6d5d64"} Oct 30 08:08:15.828719 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.828691 5894 generic.go:356] "Generic (PLEG): container finished" podID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerID="97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38" exitCode=0 Oct 30 08:08:15.828837 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.828739 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" Oct 30 08:08:15.828837 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.828771 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" event={"ID":"3ffde2fa-572a-473d-95d7-c9cf252dfb42","Type":"ContainerDied","Data":"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38"} Oct 30 08:08:15.828837 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.828813 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" event={"ID":"3ffde2fa-572a-473d-95d7-c9cf252dfb42","Type":"ContainerDied","Data":"cc9d3bc525cce9ad28a99cfe6f68051eb892a65fd3ab67904ae7363225c0d657"} Oct 30 08:08:15.838764 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.838744 5894 scope.go:122] "RemoveContainer" containerID="0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386" Oct 30 08:08:15.839048 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:15.839025 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386\": container with ID starting with 0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386 not found: ID does not exist" containerID="0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386" Oct 30 08:08:15.839125 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.839060 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386"} err="failed to get container status \"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386\": rpc error: code = NotFound desc = could not find container \"0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386\": container with ID starting with 0e67d9302c47b57181795aef965f19fcd5d0810a5a23662868f48b0c1bea8386 not found: ID does not exist" Oct 30 08:08:15.839125 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.839082 5894 scope.go:122] "RemoveContainer" containerID="97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38" Oct 30 08:08:15.852974 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.852950 5894 scope.go:122] "RemoveContainer" containerID="97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38" Oct 30 08:08:15.853313 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:15.853284 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38\": container with ID starting with 97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38 not found: ID does not exist" containerID="97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38" Oct 30 08:08:15.853420 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.853315 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38"} err="failed to get container status \"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38\": rpc error: code = NotFound desc = could not find container \"97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38\": container with ID starting with 97a783c130c88d5031dbe69f34a01d85700efafb9003d7236000d9c937c6eb38 not found: ID does not exist" Oct 30 08:08:15.884376 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.884349 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 08:08:15.891015 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.890987 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx"] Oct 30 08:08:15.931277 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.931245 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 08:08:15.934161 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.934135 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh"] Oct 30 08:08:15.951158 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:15.951126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:16.185830 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.185800 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 08:08:16.222991 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.222966 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 08:08:16.318168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.317201 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\" (UniqueName: \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\") pod \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " Oct 30 08:08:16.318168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.317464 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\") pod \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " Oct 30 08:08:16.318168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.317523 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\" (UniqueName: \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\") pod \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\" (UID: \"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e\") " Oct 30 08:08:16.318168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.317559 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\") pod \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\" (UID: \"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb\") " Oct 30 08:08:16.327206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.327127 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq" pod "71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" (UID: "71b9d412-dc54-44fd-b60f-fab3b7b2b2fb"). InnerVolumeSpecName "kube-api-access-ll8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:16.328851 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.328815 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume" pod "0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" (UID: "0f8013b5-e2ac-4bf9-8821-e0ca48f4487e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:16.330292 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.330197 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb" pod "0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" (UID: "0f8013b5-e2ac-4bf9-8821-e0ca48f4487e"). InnerVolumeSpecName "kube-api-access-44vrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:16.334986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.334951 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume" pod "71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" (UID: "71b9d412-dc54-44fd-b60f-fab3b7b2b2fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:16.398726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.398683 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-b096ec-744f64c8f4-snrdx" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.212:50052\" within 1s: context deadline exceeded" Oct 30 08:08:16.398917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.398683 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-b096ec-98f458d75-fb9wh" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.211:50052\" within 1s: context deadline exceeded" Oct 30 08:08:16.398917 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:16.398797 5894 logging.go:55] [core] [Channel #14571 SubChannel #14573]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.211:50052", ServerName: "10.134.0.211:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.211:50052: operation was canceled" Oct 30 08:08:16.398917 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:16.398801 5894 logging.go:55] [core] [Channel #14570 SubChannel #14572]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.212:50052", ServerName: "10.134.0.212:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.212:50052: operation was canceled" Oct 30 08:08:16.419352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.419318 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:16.419352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.419348 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vrb\" (UniqueName: \"kubernetes.io/projected/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e-kube-api-access-44vrb\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:16.419352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.419357 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:16.419556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.419368 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8jq\" (UniqueName: \"kubernetes.io/projected/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb-kube-api-access-ll8jq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:16.431910 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.431880 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" path="/var/lib/kubelet/pods/3ffde2fa-572a-473d-95d7-c9cf252dfb42/volumes" Oct 30 08:08:16.432310 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.432296 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" path="/var/lib/kubelet/pods/dce487fc-aaea-455b-9fe2-109bda57ec8b/volumes" Oct 30 08:08:16.699120 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.699077 5894 patch_prober.go:29] interesting pod/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 08:08:16.699288 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.699141 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 08:08:16.841097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.841061 5894 generic.go:356] "Generic (PLEG): container finished" podID="06128f7f-8928-494f-9e76-0cf26538fb51" containerID="e5c2777a7ca724df8211a98485302ab2e76af9a8bc9a7cc5ad4849cf8b25b90e" exitCode=0 Oct 30 08:08:16.841297 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.841147 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" event={"ID":"06128f7f-8928-494f-9e76-0cf26538fb51","Type":"ContainerDied","Data":"e5c2777a7ca724df8211a98485302ab2e76af9a8bc9a7cc5ad4849cf8b25b90e"} Oct 30 08:08:16.843472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.843447 5894 generic.go:356] "Generic (PLEG): container finished" podID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerID="a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781" exitCode=0 Oct 30 08:08:16.843626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.843504 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" Oct 30 08:08:16.843626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.843523 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" event={"ID":"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb","Type":"ContainerDied","Data":"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781"} Oct 30 08:08:16.843626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.843571 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj" event={"ID":"71b9d412-dc54-44fd-b60f-fab3b7b2b2fb","Type":"ContainerDied","Data":"0bb04e88843b45de6111b1550cde3b9cd3287da348a1eebc4a89c31c4d200985"} Oct 30 08:08:16.843626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.843587 5894 scope.go:122] "RemoveContainer" containerID="a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781" Oct 30 08:08:16.844926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.844837 5894 generic.go:356] "Generic (PLEG): container finished" podID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerID="82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d" exitCode=0 Oct 30 08:08:16.844926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.844902 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" Oct 30 08:08:16.845055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.844925 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" event={"ID":"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e","Type":"ContainerDied","Data":"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d"} Oct 30 08:08:16.845055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.844981 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd" event={"ID":"0f8013b5-e2ac-4bf9-8821-e0ca48f4487e","Type":"ContainerDied","Data":"5110347951ff9e6375adce95b2f582da39a343dd69723d78aebdb34f392e8ebd"} Oct 30 08:08:16.856512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.856486 5894 scope.go:122] "RemoveContainer" containerID="a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781" Oct 30 08:08:16.856804 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:16.856779 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781\": container with ID starting with a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781 not found: ID does not exist" containerID="a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781" Oct 30 08:08:16.856907 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.856813 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781"} err="failed to get container status \"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781\": rpc error: code = NotFound desc = could not find container \"a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781\": container with ID starting with a7098c5b66ee0079b34f28577739c375910f71fead528101a6a70de8488fc781 not found: ID does not exist" Oct 30 08:08:16.856907 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.856834 5894 scope.go:122] "RemoveContainer" containerID="82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d" Oct 30 08:08:16.866418 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.866400 5894 scope.go:122] "RemoveContainer" containerID="82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d" Oct 30 08:08:16.866739 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:16.866717 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d\": container with ID starting with 82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d not found: ID does not exist" containerID="82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d" Oct 30 08:08:16.866857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.866746 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d"} err="failed to get container status \"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d\": rpc error: code = NotFound desc = could not find container \"82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d\": container with ID starting with 82859934200d18793a8f6fb14c254192735099a7b7dbc29e6eca36065f95174d not found: ID does not exist" Oct 30 08:08:16.930754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.930721 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 08:08:16.934223 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.934196 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e193ee-85fd4d954b-p7rzd"] Oct 30 08:08:16.988607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.988553 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 08:08:16.991662 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:16.991614 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-e193ee-5474b874f5-5qrzj"] Oct 30 08:08:17.017071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.017045 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:17.129023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.128942 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\") pod \"b07b900c-937f-4553-a570-fe9842fd2abc\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " Oct 30 08:08:17.129023 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.129005 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\") pod \"b07b900c-937f-4553-a570-fe9842fd2abc\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " Oct 30 08:08:17.129257 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.129049 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\" (UniqueName: \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\") pod \"b07b900c-937f-4553-a570-fe9842fd2abc\" (UID: \"b07b900c-937f-4553-a570-fe9842fd2abc\") " Oct 30 08:08:17.129364 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.129338 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca" pod "b07b900c-937f-4553-a570-fe9842fd2abc" (UID: "b07b900c-937f-4553-a570-fe9842fd2abc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:17.129544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.129521 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:17.131171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.131152 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca" pod "b07b900c-937f-4553-a570-fe9842fd2abc" (UID: "b07b900c-937f-4553-a570-fe9842fd2abc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:17.131245 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.131220 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5" pod "b07b900c-937f-4553-a570-fe9842fd2abc" (UID: "b07b900c-937f-4553-a570-fe9842fd2abc"). InnerVolumeSpecName "kube-api-access-lgtt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:17.231048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.231010 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b07b900c-937f-4553-a570-fe9842fd2abc-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:17.231048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.231043 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgtt5\" (UniqueName: \"kubernetes.io/projected/b07b900c-937f-4553-a570-fe9842fd2abc-kube-api-access-lgtt5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:17.855094 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.855052 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:17.855432 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.855399 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" event={"ID":"b07b900c-937f-4553-a570-fe9842fd2abc","Type":"ContainerDied","Data":"ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59"} Oct 30 08:08:17.855614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.855456 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc451ac1657de92e80328994f346203e935e8625e96836377353f288b0b3b59" Oct 30 08:08:17.855614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:17.855413 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p" Oct 30 08:08:18.029137 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.029112 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:18.142697 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.142563 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\") pod \"06128f7f-8928-494f-9e76-0cf26538fb51\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " Oct 30 08:08:18.142885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.142730 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\" (UniqueName: \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\") pod \"06128f7f-8928-494f-9e76-0cf26538fb51\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " Oct 30 08:08:18.142885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.142772 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\") pod \"06128f7f-8928-494f-9e76-0cf26538fb51\" (UID: \"06128f7f-8928-494f-9e76-0cf26538fb51\") " Oct 30 08:08:18.143209 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.143180 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca" pod "06128f7f-8928-494f-9e76-0cf26538fb51" (UID: "06128f7f-8928-494f-9e76-0cf26538fb51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:18.143358 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.143337 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.145004 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.144969 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca" pod "06128f7f-8928-494f-9e76-0cf26538fb51" (UID: "06128f7f-8928-494f-9e76-0cf26538fb51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:18.145123 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.145065 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt" pod "06128f7f-8928-494f-9e76-0cf26538fb51" (UID: "06128f7f-8928-494f-9e76-0cf26538fb51"). InnerVolumeSpecName "kube-api-access-c9mtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:18.244212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.244169 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06128f7f-8928-494f-9e76-0cf26538fb51-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.244212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.244204 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9mtt\" (UniqueName: \"kubernetes.io/projected/06128f7f-8928-494f-9e76-0cf26538fb51-kube-api-access-c9mtt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.414041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.414014 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 08:08:18.430795 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.430762 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" path="/var/lib/kubelet/pods/0f8013b5-e2ac-4bf9-8821-e0ca48f4487e/volumes" Oct 30 08:08:18.431325 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.431304 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" path="/var/lib/kubelet/pods/71b9d412-dc54-44fd-b60f-fab3b7b2b2fb/volumes" Oct 30 08:08:18.546674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546619 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.546827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546690 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.546827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546743 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.546827 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546766 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546839 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546881 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546924 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546968 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:18.547032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.546975 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547073 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:18.547317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547121 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\") pod \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\" (UID: \"c0bb61db-2d23-4aac-9ade-184a5e5ed99f\") " Oct 30 08:08:18.547317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547152 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:18.547317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547275 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:18.547317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547306 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:18.547607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547556 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.547607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547572 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.547607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547583 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.547607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547597 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.547607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.547605 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.549289 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.549264 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:18.549451 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.549321 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "kube-api-access-jnsjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:18.549525 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.549501 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:08:18.549720 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.549697 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo" pod "c0bb61db-2d23-4aac-9ade-184a5e5ed99f" (UID: "c0bb61db-2d23-4aac-9ade-184a5e5ed99f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:08:18.649097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.649019 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnsjv\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-kube-api-access-jnsjv\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.649097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.649051 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.649097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.649062 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.649097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.649070 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0bb61db-2d23-4aac-9ade-184a5e5ed99f-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.732483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.732457 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 08:08:18.749544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.749517 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\" (UniqueName: \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\") pod \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " Oct 30 08:08:18.749715 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.749564 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\") pod \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\" (UID: \"2cc28a87-1187-4860-bad5-ff4673b1f1a3\") " Oct 30 08:08:18.750040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.750011 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume" pod "2cc28a87-1187-4860-bad5-ff4673b1f1a3" (UID: "2cc28a87-1187-4860-bad5-ff4673b1f1a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:18.751782 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.751759 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr" pod "2cc28a87-1187-4860-bad5-ff4673b1f1a3" (UID: "2cc28a87-1187-4860-bad5-ff4673b1f1a3"). InnerVolumeSpecName "kube-api-access-phwqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:18.851172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.851129 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phwqr\" (UniqueName: \"kubernetes.io/projected/2cc28a87-1187-4860-bad5-ff4673b1f1a3-kube-api-access-phwqr\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.851172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.851164 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cc28a87-1187-4860-bad5-ff4673b1f1a3-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:18.867077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.867040 5894 generic.go:356] "Generic (PLEG): container finished" podID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerID="55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb" exitCode=0 Oct 30 08:08:18.867234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.867090 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" Oct 30 08:08:18.867234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.867120 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerDied","Data":"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb"} Oct 30 08:08:18.867234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.867173 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj" event={"ID":"c0bb61db-2d23-4aac-9ade-184a5e5ed99f","Type":"ContainerDied","Data":"33a15e3be9527ed2266d09ef40dd9cff0a5430a5099b33cec2eba2e387a31188"} Oct 30 08:08:18.867234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.867205 5894 scope.go:122] "RemoveContainer" containerID="55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb" Oct 30 08:08:18.868889 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.868865 5894 generic.go:356] "Generic (PLEG): container finished" podID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerID="3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd" exitCode=0 Oct 30 08:08:18.869003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.868988 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" Oct 30 08:08:18.869080 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.868988 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" event={"ID":"2cc28a87-1187-4860-bad5-ff4673b1f1a3","Type":"ContainerDied","Data":"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd"} Oct 30 08:08:18.869155 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.869123 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v" event={"ID":"2cc28a87-1187-4860-bad5-ff4673b1f1a3","Type":"ContainerDied","Data":"b9ec47c4902549472afc495a850d0d63a09b54106faa7178a19eaa7a4b08f8b6"} Oct 30 08:08:18.870977 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.870950 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" Oct 30 08:08:18.871101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.870987 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr" event={"ID":"06128f7f-8928-494f-9e76-0cf26538fb51","Type":"ContainerDied","Data":"6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a"} Oct 30 08:08:18.871101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.871043 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c2cdc736890179393f61073587cf4e492aa07eed54da4e090c7a8847625562a" Oct 30 08:08:18.880918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.880899 5894 scope.go:122] "RemoveContainer" containerID="2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910" Oct 30 08:08:18.890854 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.890834 5894 scope.go:122] "RemoveContainer" containerID="55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb" Oct 30 08:08:18.891173 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:18.891141 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb\": container with ID starting with 55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb not found: ID does not exist" containerID="55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb" Oct 30 08:08:18.891232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.891183 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb"} err="failed to get container status \"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb\": rpc error: code = NotFound desc = could not find container \"55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb\": container with ID starting with 55c5ff6b229ef871b24927e9f87218312ef0bb567dceb9db2bb53dac632ba3fb not found: ID does not exist" Oct 30 08:08:18.891232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.891202 5894 scope.go:122] "RemoveContainer" containerID="2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910" Oct 30 08:08:18.891479 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:18.891459 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910\": container with ID starting with 2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910 not found: ID does not exist" containerID="2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910" Oct 30 08:08:18.891610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.891491 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910"} err="failed to get container status \"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910\": rpc error: code = NotFound desc = could not find container \"2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910\": container with ID starting with 2b3088090a19f8f4eb0fe7006d305e57af591344bd532a78b4653fc542c86910 not found: ID does not exist" Oct 30 08:08:18.891610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.891516 5894 scope.go:122] "RemoveContainer" containerID="3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd" Oct 30 08:08:18.900975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.900954 5894 scope.go:122] "RemoveContainer" containerID="3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd" Oct 30 08:08:18.901249 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:18.901229 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd\": container with ID starting with 3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd not found: ID does not exist" containerID="3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd" Oct 30 08:08:18.901328 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.901260 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd"} err="failed to get container status \"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd\": rpc error: code = NotFound desc = could not find container \"3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd\": container with ID starting with 3aa691b1a4cb763c7a0d0eda4d83ab81f742a4f1fc551b1bd8a53c8d11bbc6bd not found: ID does not exist" Oct 30 08:08:18.944731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.944675 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 08:08:18.948392 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.948368 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-d89593-67488d59f8-n5z4v"] Oct 30 08:08:18.989274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.989236 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 08:08:18.993214 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:18.993184 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e193ee-openshift-default-bf76f6977-q52nj"] Oct 30 08:08:19.666233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:19.666107 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:20.432008 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:20.431970 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" path="/var/lib/kubelet/pods/2cc28a87-1187-4860-bad5-ff4673b1f1a3/volumes" Oct 30 08:08:20.432462 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:20.432447 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" path="/var/lib/kubelet/pods/c0bb61db-2d23-4aac-9ade-184a5e5ed99f/volumes" Oct 30 08:08:20.981340 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:20.981302 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:21.093077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.093038 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:21.342142 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.342108 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:21.454839 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.454808 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:21.892336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892302 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt"] Oct 30 08:08:21.892926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892905 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:21.892926 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892926 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892945 5894 cpu_manager.go:414] "containerMap: removing container" podUID="b07b900c-937f-4553-a570-fe9842fd2abc" containerName="revoke-keys" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892952 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b07b900c-937f-4553-a570-fe9842fd2abc" containerName="revoke-keys" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892966 5894 cpu_manager.go:414] "containerMap: removing container" podUID="06128f7f-8928-494f-9e76-0cf26538fb51" containerName="revoke-keys" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892974 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06128f7f-8928-494f-9e76-0cf26538fb51" containerName="revoke-keys" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.892994 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893001 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893021 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893029 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893051 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893058 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893070 5894 cpu_manager.go:414] "containerMap: removing container" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893077 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893088 5894 cpu_manager.go:414] "containerMap: removing container" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" Oct 30 08:08:21.893113 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893097 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893253 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893267 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="b07b900c-937f-4553-a570-fe9842fd2abc" containerName="revoke-keys" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893278 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="06128f7f-8928-494f-9e76-0cf26538fb51" containerName="revoke-keys" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893287 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffde2fa-572a-473d-95d7-c9cf252dfb42" containerName="payload-processing" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893300 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8013b5-e2ac-4bf9-8821-e0ca48f4487e" containerName="payload-processing" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893311 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cc28a87-1187-4860-bad5-ff4673b1f1a3" containerName="payload-processing" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893320 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce487fc-aaea-455b-9fe2-109bda57ec8b" containerName="payload-pre-processing" Oct 30 08:08:21.893905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.893330 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b9d412-dc54-44fd-b60f-fab3b7b2b2fb" containerName="payload-pre-processing" Oct 30 08:08:21.897543 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.897521 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:21.915056 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.915017 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt"] Oct 30 08:08:21.926314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.926284 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:08:21.926467 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.926339 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:08:21.980403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.980369 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:08:21.980778 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.980759 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:08:21.988439 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.988414 5894 scope.go:122] "RemoveContainer" containerID="56dad326f4e01c3af7bd8703985f5d2e62d8eff121bc12ac8d475f5ed38ddb12" Oct 30 08:08:21.991930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.991910 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:21.992019 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.991979 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:21.992141 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.992123 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc6s\" (UniqueName: \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:21.998698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:21.998677 5894 scope.go:122] "RemoveContainer" containerID="b8f7d2a53d41005144292c07d4eb4e235c7bcc741cb93981466a852931f548ee" Oct 30 08:08:22.093191 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.093161 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.093355 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.093290 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc6s\" (UniqueName: \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.093404 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.093377 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.093917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.093896 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.095882 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.095857 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.119734 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.119704 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc6s\" (UniqueName: \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.212621 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.212535 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:22.518555 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:22.518517 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d7f2059_a40a_43fb_ab48_f60c712ed336.slice/crio-b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8 WatchSource:0}: Error finding container b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8: Status 404 returned error can't find the container with id b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8 Oct 30 08:08:22.520802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.520425 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt"] Oct 30 08:08:22.906067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.906028 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" event={"ID":"2d7f2059-a40a-43fb-ab48-f60c712ed336","Type":"ContainerStarted","Data":"482cb73c99a343b9a96ebd98af281cb0b6a34025fc2eb565ba6c861850ce8242"} Oct 30 08:08:22.906455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.906088 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" event={"ID":"2d7f2059-a40a-43fb-ab48-f60c712ed336","Type":"ContainerStarted","Data":"b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8"} Oct 30 08:08:22.951363 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:22.951311 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" podStartSLOduration=1.95129589 podStartE2EDuration="1.95129589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:08:22.950217321 +0000 UTC m=+4266.641134760" watchObservedRunningTime="2026-10-30 08:08:22.95129589 +0000 UTC m=+4266.642213323" Oct 30 08:08:23.031604 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:23.031511 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v"] Oct 30 08:08:23.034557 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:23.034521 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac3f2a68-xrq4v"] Oct 30 08:08:23.245607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:23.245571 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:23.918439 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:23.918403 5894 generic.go:356] "Generic (PLEG): container finished" podID="2d7f2059-a40a-43fb-ab48-f60c712ed336" containerID="482cb73c99a343b9a96ebd98af281cb0b6a34025fc2eb565ba6c861850ce8242" exitCode=0 Oct 30 08:08:23.918680 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:23.918499 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" event={"ID":"2d7f2059-a40a-43fb-ab48-f60c712ed336","Type":"ContainerDied","Data":"482cb73c99a343b9a96ebd98af281cb0b6a34025fc2eb565ba6c861850ce8242"} Oct 30 08:08:24.431729 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:24.431695 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ffc6ad2-a607-4e99-8991-675bfd3a0ea3" path="/var/lib/kubelet/pods/4ffc6ad2-a607-4e99-8991-675bfd3a0ea3/volumes" Oct 30 08:08:24.662713 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:24.662669 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:24.776982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:24.776894 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:25.044745 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.044613 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:25.074339 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.074312 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:25.148077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.146199 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\") pod \"2d7f2059-a40a-43fb-ab48-f60c712ed336\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " Oct 30 08:08:25.148077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.146273 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\" (UniqueName: \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\") pod \"2d7f2059-a40a-43fb-ab48-f60c712ed336\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " Oct 30 08:08:25.148077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.146515 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\") pod \"2d7f2059-a40a-43fb-ab48-f60c712ed336\" (UID: \"2d7f2059-a40a-43fb-ab48-f60c712ed336\") " Oct 30 08:08:25.148602 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.148553 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca" pod "2d7f2059-a40a-43fb-ab48-f60c712ed336" (UID: "2d7f2059-a40a-43fb-ab48-f60c712ed336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:25.165291 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.165204 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s" pod "2d7f2059-a40a-43fb-ab48-f60c712ed336" (UID: "2d7f2059-a40a-43fb-ab48-f60c712ed336"). InnerVolumeSpecName "kube-api-access-pdc6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:25.184111 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.184068 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca" pod "2d7f2059-a40a-43fb-ab48-f60c712ed336" (UID: "2d7f2059-a40a-43fb-ab48-f60c712ed336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:25.248110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.248074 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:25.248110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.248105 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d7f2059-a40a-43fb-ab48-f60c712ed336-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:25.248110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.248116 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdc6s\" (UniqueName: \"kubernetes.io/projected/2d7f2059-a40a-43fb-ab48-f60c712ed336-kube-api-access-pdc6s\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:25.258766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.258736 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 08:08:25.293518 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.293471 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 08:08:25.410149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.410060 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:25.533264 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.533208 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:25.934476 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.934440 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" Oct 30 08:08:25.934722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.934439 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt" event={"ID":"2d7f2059-a40a-43fb-ab48-f60c712ed336","Type":"ContainerDied","Data":"b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8"} Oct 30 08:08:25.934722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:25.934570 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6638b6c4eb61482173530dcdc06a585a5da82682b89922159ea77f2c45b21e8" Oct 30 08:08:26.960296 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:26.960251 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:28.354788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:28.354750 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:28.846183 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:28.846131 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:28.960401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:28.960360 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:28.979975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:28.979943 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr"] Oct 30 08:08:29.005660 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:29.004048 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-0f76b0-vncpr"] Oct 30 08:08:29.488621 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:29.488584 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:30.259909 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:30.259870 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" containerID="cri-o://a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885" gracePeriod=25 Oct 30 08:08:30.294483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:30.294433 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" containerID="cri-o://8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44" gracePeriod=25 Oct 30 08:08:30.439167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:30.439120 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06128f7f-8928-494f-9e76-0cf26538fb51" path="/var/lib/kubelet/pods/06128f7f-8928-494f-9e76-0cf26538fb51/volumes" Oct 30 08:08:30.734097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:30.734061 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:32.428790 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:32.428751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:32.557381 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:32.557344 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:33.282210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:33.282173 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:33.282210 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:33.282185 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:34.006571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.006522 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:08:34.033428 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.033395 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf"] Oct 30 08:08:34.036436 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.036401 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3e0bb0-8svcf"] Oct 30 08:08:34.086743 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.086711 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:34.334012 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.333976 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:34.431171 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.431133 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0703ff75-2966-42c9-ae19-670b4cdc810c" path="/var/lib/kubelet/pods/0703ff75-2966-42c9-ae19-670b4cdc810c/volumes" Oct 30 08:08:34.485557 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:34.485522 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:34.828252 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:34.828215 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:08:35.668303 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:35.668267 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:36.231415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.231379 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:36.359405 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.359370 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt"] Oct 30 08:08:36.374040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.374008 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-054ef1-4n6kt"] Oct 30 08:08:36.379309 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.379277 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:36.431118 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.431083 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d7f2059-a40a-43fb-ab48-f60c712ed336" path="/var/lib/kubelet/pods/2d7f2059-a40a-43fb-ab48-f60c712ed336/volumes" Oct 30 08:08:36.643275 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.643237 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:36.763075 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.763029 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:36.882164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:36.882126 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:37.000352 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.000246 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:37.530413 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.530376 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms"] Oct 30 08:08:37.530937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.530922 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:37.531003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.530938 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:37.531003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.530957 5894 cpu_manager.go:414] "containerMap: removing container" podUID="2d7f2059-a40a-43fb-ab48-f60c712ed336" containerName="revoke-keys" Oct 30 08:08:37.531003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.530962 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d7f2059-a40a-43fb-ab48-f60c712ed336" containerName="revoke-keys" Oct 30 08:08:37.531108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.531043 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d7f2059-a40a-43fb-ab48-f60c712ed336" containerName="revoke-keys" Oct 30 08:08:37.531108 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.531060 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bb61db-2d23-4aac-9ade-184a5e5ed99f" containerName="istio-proxy" Oct 30 08:08:37.535311 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.535292 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.553307 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.553275 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms"] Oct 30 08:08:37.564919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.564885 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:08:37.564919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.564910 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:08:37.566423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.566396 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:08:37.566590 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.566511 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:08:37.716881 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.716836 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.717084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.716887 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5sq\" (UniqueName: \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.717084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.717008 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.818607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.818501 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.818607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.818553 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5sq\" (UniqueName: \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.818607 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.818597 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.819431 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.819408 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.821081 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.821053 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.844668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.844618 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5sq\" (UniqueName: \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.849493 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.849468 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:37.988173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:37.988139 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:38.157847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.156754 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms"] Oct 30 08:08:38.310580 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.310542 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:38.576169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.576114 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:38.710112 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.710007 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:38.710726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.710703 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p"] Oct 30 08:08:38.724601 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.724567 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-2e8b2a-69m9p"] Oct 30 08:08:38.824820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:38.824784 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:39.030600 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:39.030567 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" event={"ID":"25a6b14e-e024-468f-afb4-13133118bebd","Type":"ContainerStarted","Data":"4de2ec5e7416b533a0ee6225aabc90f579163676672467e7dcb16fb35f647e42"} Oct 30 08:08:39.030797 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:39.030619 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" event={"ID":"25a6b14e-e024-468f-afb4-13133118bebd","Type":"ContainerStarted","Data":"fdf3a88a2aeae1d2cda59d0449e3c4bdda1c12a295abe74a412f8b64b84a1fe1"} Oct 30 08:08:39.078056 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:39.078000 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" podStartSLOduration=2.077984962 podStartE2EDuration="2.077984962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:08:39.075606275 +0000 UTC m=+4282.766523712" watchObservedRunningTime="2026-10-30 08:08:39.077984962 +0000 UTC m=+4282.768902399" Oct 30 08:08:39.198985 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:39.198949 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:40.040011 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:40.039974 5894 generic.go:356] "Generic (PLEG): container finished" podID="25a6b14e-e024-468f-afb4-13133118bebd" containerID="4de2ec5e7416b533a0ee6225aabc90f579163676672467e7dcb16fb35f647e42" exitCode=0 Oct 30 08:08:40.040489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:40.040059 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" event={"ID":"25a6b14e-e024-468f-afb4-13133118bebd","Type":"ContainerDied","Data":"4de2ec5e7416b533a0ee6225aabc90f579163676672467e7dcb16fb35f647e42"} Oct 30 08:08:40.431535 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:40.431436 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b07b900c-937f-4553-a570-fe9842fd2abc" path="/var/lib/kubelet/pods/b07b900c-937f-4553-a570-fe9842fd2abc/volumes" Oct 30 08:08:40.625712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:40.625676 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:41.204214 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.204184 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:41.257021 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.256988 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\" (UniqueName: \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\") pod \"25a6b14e-e024-468f-afb4-13133118bebd\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " Oct 30 08:08:41.257225 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.257076 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\") pod \"25a6b14e-e024-468f-afb4-13133118bebd\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " Oct 30 08:08:41.257225 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.257136 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\") pod \"25a6b14e-e024-468f-afb4-13133118bebd\" (UID: \"25a6b14e-e024-468f-afb4-13133118bebd\") " Oct 30 08:08:41.257594 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.257565 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca" pod "25a6b14e-e024-468f-afb4-13133118bebd" (UID: "25a6b14e-e024-468f-afb4-13133118bebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:41.259393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.259368 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq" pod "25a6b14e-e024-468f-afb4-13133118bebd" (UID: "25a6b14e-e024-468f-afb4-13133118bebd"). InnerVolumeSpecName "kube-api-access-pl5sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:41.259657 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.259562 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca" pod "25a6b14e-e024-468f-afb4-13133118bebd" (UID: "25a6b14e-e024-468f-afb4-13133118bebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:41.358595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.358512 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl5sq\" (UniqueName: \"kubernetes.io/projected/25a6b14e-e024-468f-afb4-13133118bebd-kube-api-access-pl5sq\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:41.358595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.358543 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:41.358595 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:41.358554 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25a6b14e-e024-468f-afb4-13133118bebd-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:42.056369 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:42.056338 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" Oct 30 08:08:42.056555 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:42.056369 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms" event={"ID":"25a6b14e-e024-468f-afb4-13133118bebd","Type":"ContainerDied","Data":"fdf3a88a2aeae1d2cda59d0449e3c4bdda1c12a295abe74a412f8b64b84a1fe1"} Oct 30 08:08:42.056555 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:42.056446 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf3a88a2aeae1d2cda59d0449e3c4bdda1c12a295abe74a412f8b64b84a1fe1" Oct 30 08:08:42.117841 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:42.117803 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a6b14e_e024_468f_afb4_13133118bebd.slice/crio-fdf3a88a2aeae1d2cda59d0449e3c4bdda1c12a295abe74a412f8b64b84a1fe1\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a6b14e_e024_468f_afb4_13133118bebd.slice\": RecentStats: unable to find data in memory cache]" Oct 30 08:08:42.315167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:42.315090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:43.282605 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:43.282566 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:43.282605 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:43.282588 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:08:43.407828 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:43.407782 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:43.529817 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:43.529775 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:43.824478 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:43.824444 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:44.093378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.093289 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:44.331240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.331202 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:44.647666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.647603 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b"] Oct 30 08:08:44.648169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.648152 5894 cpu_manager.go:414] "containerMap: removing container" podUID="25a6b14e-e024-468f-afb4-13133118bebd" containerName="revoke-keys" Oct 30 08:08:44.648232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.648171 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a6b14e-e024-468f-afb4-13133118bebd" containerName="revoke-keys" Oct 30 08:08:44.648325 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.648286 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a6b14e-e024-468f-afb4-13133118bebd" containerName="revoke-keys" Oct 30 08:08:44.652046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.652028 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.674608 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.674568 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:08:44.676788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.676759 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b"] Oct 30 08:08:44.687172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.687135 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:08:44.687172 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.687166 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:08:44.687436 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.687182 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:08:44.696964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.696928 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84tx\" (UniqueName: \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.697134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.697108 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.697279 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.697219 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.798475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.798439 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84tx\" (UniqueName: \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.798750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.798562 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.798750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.798608 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.799375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.799354 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.801119 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.801098 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.825537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.825502 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84tx\" (UniqueName: \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\") pod \"maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:44.966064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:44.965979 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:45.036049 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:45.036008 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd"] Oct 30 08:08:45.039774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:45.039701 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaf4b0f5-7cfmd"] Oct 30 08:08:45.280293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:45.280249 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b"] Oct 30 08:08:45.280881 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:45.280850 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b178415_02c9_444c_ae8c_8b907183c3c1.slice/crio-9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640 WatchSource:0}: Error finding container 9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640: Status 404 returned error can't find the container with id 9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640 Oct 30 08:08:46.088694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.088626 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" event={"ID":"9b178415-02c9-444c-ae8c-8b907183c3c1","Type":"ContainerStarted","Data":"b4d7af98827bd585e3097b9b056dbcddc18a54dc0bf0464617a76ffb1478f93a"} Oct 30 08:08:46.089182 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.088713 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" event={"ID":"9b178415-02c9-444c-ae8c-8b907183c3c1","Type":"ContainerStarted","Data":"9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640"} Oct 30 08:08:46.134688 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.134608 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" podStartSLOduration=2.134594032 podStartE2EDuration="2.134594032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:08:46.132963752 +0000 UTC m=+4289.823881190" watchObservedRunningTime="2026-10-30 08:08:46.134594032 +0000 UTC m=+4289.825511466" Oct 30 08:08:46.224530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.224489 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:46.431932 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.431855 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61ef24b-724b-41b8-acb3-130099bcdb01" path="/var/lib/kubelet/pods/c61ef24b-724b-41b8-acb3-130099bcdb01/volumes" Oct 30 08:08:46.456014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.455983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:46.572489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.572452 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:46.788506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:46.788469 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:47.014929 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.014894 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:08:47.096862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.096774 5894 generic.go:356] "Generic (PLEG): container finished" podID="9b178415-02c9-444c-ae8c-8b907183c3c1" containerID="b4d7af98827bd585e3097b9b056dbcddc18a54dc0bf0464617a76ffb1478f93a" exitCode=0 Oct 30 08:08:47.096862 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.096825 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" event={"ID":"9b178415-02c9-444c-ae8c-8b907183c3c1","Type":"ContainerDied","Data":"b4d7af98827bd585e3097b9b056dbcddc18a54dc0bf0464617a76ffb1478f93a"} Oct 30 08:08:47.339917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.339871 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh"] Oct 30 08:08:47.344357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.344337 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.362403 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.362373 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh"] Oct 30 08:08:47.437552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.437511 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkhzj\" (UniqueName: \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.437552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.437561 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.437821 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.437724 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.538895 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.538850 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.539114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.539010 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkhzj\" (UniqueName: \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.539114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.539044 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.539793 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.539772 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.541486 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.541460 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.566416 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.566379 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkhzj\" (UniqueName: \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.659286 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.659182 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:47.854863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.854817 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:47.982092 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:47.982058 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac068175_77c4_4983_ab26_1a95f4bbbb2a.slice/crio-acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651 WatchSource:0}: Error finding container acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651: Status 404 returned error can't find the container with id acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651 Oct 30 08:08:47.982400 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:47.982371 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh"] Oct 30 08:08:48.059533 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.059487 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:48.105058 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.105024 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" event={"ID":"ac068175-77c4-4983-ab26-1a95f4bbbb2a","Type":"ContainerStarted","Data":"acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651"} Oct 30 08:08:48.178038 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.177983 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:48.260872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.260846 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:48.356960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.356847 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\" (UniqueName: \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\") pod \"9b178415-02c9-444c-ae8c-8b907183c3c1\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " Oct 30 08:08:48.356960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.356961 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\") pod \"9b178415-02c9-444c-ae8c-8b907183c3c1\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " Oct 30 08:08:48.357232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.357015 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\") pod \"9b178415-02c9-444c-ae8c-8b907183c3c1\" (UID: \"9b178415-02c9-444c-ae8c-8b907183c3c1\") " Oct 30 08:08:48.357390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.357361 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca" pod "9b178415-02c9-444c-ae8c-8b907183c3c1" (UID: "9b178415-02c9-444c-ae8c-8b907183c3c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:48.357510 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.357477 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:48.359711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.359684 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx" pod "9b178415-02c9-444c-ae8c-8b907183c3c1" (UID: "9b178415-02c9-444c-ae8c-8b907183c3c1"). InnerVolumeSpecName "kube-api-access-l84tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:48.359986 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.359965 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca" pod "9b178415-02c9-444c-ae8c-8b907183c3c1" (UID: "9b178415-02c9-444c-ae8c-8b907183c3c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:48.458626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.458587 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b178415-02c9-444c-ae8c-8b907183c3c1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:48.458626 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:48.458623 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84tx\" (UniqueName: \"kubernetes.io/projected/9b178415-02c9-444c-ae8c-8b907183c3c1-kube-api-access-l84tx\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:49.115397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.115360 5894 generic.go:356] "Generic (PLEG): container finished" podID="ac068175-77c4-4983-ab26-1a95f4bbbb2a" containerID="abe0465065e58fe9227cae5bb38dac6460f4477f23f94003745839e232c878d5" exitCode=0 Oct 30 08:08:49.115825 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.115433 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" event={"ID":"ac068175-77c4-4983-ab26-1a95f4bbbb2a","Type":"ContainerDied","Data":"abe0465065e58fe9227cae5bb38dac6460f4477f23f94003745839e232c878d5"} Oct 30 08:08:49.117152 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.117126 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" event={"ID":"9b178415-02c9-444c-ae8c-8b907183c3c1","Type":"ContainerDied","Data":"9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640"} Oct 30 08:08:49.117302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.117165 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c69d6f58016e81289217ccb6faf7eee2ff474086e61ebc2e9718ca5f9ee2640" Oct 30 08:08:49.117302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.117167 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b" Oct 30 08:08:49.366996 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.366949 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:49.921496 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:49.921460 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:50.414236 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.414215 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:50.483269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.483238 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\" (UniqueName: \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\") pod \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " Oct 30 08:08:50.483446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.483280 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\") pod \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " Oct 30 08:08:50.483446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.483404 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\") pod \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\" (UID: \"ac068175-77c4-4983-ab26-1a95f4bbbb2a\") " Oct 30 08:08:50.483823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.483794 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca" pod "ac068175-77c4-4983-ab26-1a95f4bbbb2a" (UID: "ac068175-77c4-4983-ab26-1a95f4bbbb2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:50.485668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.485615 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj" pod "ac068175-77c4-4983-ab26-1a95f4bbbb2a" (UID: "ac068175-77c4-4983-ab26-1a95f4bbbb2a"). InnerVolumeSpecName "kube-api-access-dkhzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:50.486162 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.486137 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca" pod "ac068175-77c4-4983-ab26-1a95f4bbbb2a" (UID: "ac068175-77c4-4983-ab26-1a95f4bbbb2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:08:50.580680 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.580649 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 08:08:50.584905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.584874 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkhzj\" (UniqueName: \"kubernetes.io/projected/ac068175-77c4-4983-ab26-1a95f4bbbb2a-kube-api-access-dkhzj\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.584905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.584908 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.585090 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.584922 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac068175-77c4-4983-ab26-1a95f4bbbb2a-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.588048 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.588029 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 08:08:50.685308 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685270 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\" (UniqueName: \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\") pod \"a27d8357-d617-4303-8e82-78ff0d6c38fd\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " Oct 30 08:08:50.685530 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685430 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\") pod \"225e48aa-8df4-4f7e-ad59-75448a449c6e\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " Oct 30 08:08:50.685610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685550 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\") pod \"a27d8357-d617-4303-8e82-78ff0d6c38fd\" (UID: \"a27d8357-d617-4303-8e82-78ff0d6c38fd\") " Oct 30 08:08:50.685610 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685588 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\" (UniqueName: \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\") pod \"225e48aa-8df4-4f7e-ad59-75448a449c6e\" (UID: \"225e48aa-8df4-4f7e-ad59-75448a449c6e\") " Oct 30 08:08:50.685956 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685932 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume" pod "a27d8357-d617-4303-8e82-78ff0d6c38fd" (UID: "a27d8357-d617-4303-8e82-78ff0d6c38fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:50.685956 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.685928 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume" pod "225e48aa-8df4-4f7e-ad59-75448a449c6e" (UID: "225e48aa-8df4-4f7e-ad59-75448a449c6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:08:50.687797 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.687769 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk" pod "225e48aa-8df4-4f7e-ad59-75448a449c6e" (UID: "225e48aa-8df4-4f7e-ad59-75448a449c6e"). InnerVolumeSpecName "kube-api-access-g2jpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:50.688104 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.688081 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q" pod "a27d8357-d617-4303-8e82-78ff0d6c38fd" (UID: "a27d8357-d617-4303-8e82-78ff0d6c38fd"). InnerVolumeSpecName "kube-api-access-p299q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:08:50.787285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.787251 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a27d8357-d617-4303-8e82-78ff0d6c38fd-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.787285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.787280 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2jpk\" (UniqueName: \"kubernetes.io/projected/225e48aa-8df4-4f7e-ad59-75448a449c6e-kube-api-access-g2jpk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.787285 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.787290 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p299q\" (UniqueName: \"kubernetes.io/projected/a27d8357-d617-4303-8e82-78ff0d6c38fd-kube-api-access-p299q\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:50.787539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:50.787301 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225e48aa-8df4-4f7e-ad59-75448a449c6e-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:08:51.134407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.134304 5894 generic.go:356] "Generic (PLEG): container finished" podID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerID="8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44" exitCode=0 Oct 30 08:08:51.134407 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.134358 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" Oct 30 08:08:51.134696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.134396 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" event={"ID":"225e48aa-8df4-4f7e-ad59-75448a449c6e","Type":"ContainerDied","Data":"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44"} Oct 30 08:08:51.134696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.134452 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg" event={"ID":"225e48aa-8df4-4f7e-ad59-75448a449c6e","Type":"ContainerDied","Data":"f8bb660c41bf3a7a899f427630f86a4114f246c90fa22e981328bd9b22bd930a"} Oct 30 08:08:51.134696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.134479 5894 scope.go:122] "RemoveContainer" containerID="8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44" Oct 30 08:08:51.136014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.135986 5894 generic.go:356] "Generic (PLEG): container finished" podID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerID="a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885" exitCode=0 Oct 30 08:08:51.136101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.136015 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" event={"ID":"a27d8357-d617-4303-8e82-78ff0d6c38fd","Type":"ContainerDied","Data":"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885"} Oct 30 08:08:51.136101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.136036 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" Oct 30 08:08:51.136101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.136067 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl" event={"ID":"a27d8357-d617-4303-8e82-78ff0d6c38fd","Type":"ContainerDied","Data":"fbf17829b890228d09a158891dd27818ad934a7f71f6ce11ff0eee0bc425af8c"} Oct 30 08:08:51.137748 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.137714 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" event={"ID":"ac068175-77c4-4983-ab26-1a95f4bbbb2a","Type":"ContainerDied","Data":"acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651"} Oct 30 08:08:51.137748 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.137731 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh" Oct 30 08:08:51.137948 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.137759 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acac02c2604796a8d4b306b8f86608a13f0cdfa603c702a11a24ae17f65a9651" Oct 30 08:08:51.147737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.147716 5894 scope.go:122] "RemoveContainer" containerID="8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44" Oct 30 08:08:51.148052 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:51.148029 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44\": container with ID starting with 8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44 not found: ID does not exist" containerID="8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44" Oct 30 08:08:51.148136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.148061 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44"} err="failed to get container status \"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44\": rpc error: code = NotFound desc = could not find container \"8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44\": container with ID starting with 8580d6751027c230132038cc7a4308de2ea2c4e95a876bd5b18fbc2aad52bb44 not found: ID does not exist" Oct 30 08:08:51.148136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.148086 5894 scope.go:122] "RemoveContainer" containerID="a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885" Oct 30 08:08:51.169844 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.169818 5894 scope.go:122] "RemoveContainer" containerID="a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885" Oct 30 08:08:51.170252 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:08:51.170203 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885\": container with ID starting with a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885 not found: ID does not exist" containerID="a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885" Oct 30 08:08:51.170380 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.170256 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885"} err="failed to get container status \"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885\": rpc error: code = NotFound desc = could not find container \"a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885\": container with ID starting with a07b3765c96a989d80e46ae366f351e4a4c931d06c27fc229ec9fe471259e885 not found: ID does not exist" Oct 30 08:08:51.203554 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.203519 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 08:08:51.208237 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.208206 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0f76b0-86b95689d4-4rzjl"] Oct 30 08:08:51.244654 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.244599 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 08:08:51.248940 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.248913 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-0f76b0-f7574d8dc-prfdg"] Oct 30 08:08:51.400124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.400025 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:51.979505 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:51.979447 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:52.105393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.105357 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms"] Oct 30 08:08:52.108144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.108090 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:52.119378 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.119344 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bf563a-tcxms"] Oct 30 08:08:52.232797 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.232707 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:52.432750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.432719 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" path="/var/lib/kubelet/pods/225e48aa-8df4-4f7e-ad59-75448a449c6e/volumes" Oct 30 08:08:52.433180 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.433167 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a6b14e-e024-468f-afb4-13133118bebd" path="/var/lib/kubelet/pods/25a6b14e-e024-468f-afb4-13133118bebd/volumes" Oct 30 08:08:52.433524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.433513 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" path="/var/lib/kubelet/pods/a27d8357-d617-4303-8e82-78ff0d6c38fd/volumes" Oct 30 08:08:52.618989 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:52.618954 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:54.034018 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:54.033987 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:55.038175 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:55.038137 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q"] Oct 30 08:08:55.052499 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:55.052464 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974b208a-bxh4q"] Oct 30 08:08:55.222960 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:55.222897 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:55.338754 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:55.338651 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:55.743348 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:55.743233 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:56.431869 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:56.431831 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08226c3d-b049-4226-a27d-7f555fabc3ff" path="/var/lib/kubelet/pods/08226c3d-b049-4226-a27d-7f555fabc3ff/volumes" Oct 30 08:08:57.684939 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:57.684902 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:57.953062 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:57.952972 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:58.070484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:58.070449 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:59.133167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.133119 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs"] Oct 30 08:08:59.134427 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134394 5894 cpu_manager.go:414] "containerMap: removing container" podUID="ac068175-77c4-4983-ab26-1a95f4bbbb2a" containerName="revoke-keys" Oct 30 08:08:59.134427 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134426 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac068175-77c4-4983-ab26-1a95f4bbbb2a" containerName="revoke-keys" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134466 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9b178415-02c9-444c-ae8c-8b907183c3c1" containerName="revoke-keys" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134473 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b178415-02c9-444c-ae8c-8b907183c3c1" containerName="revoke-keys" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134508 5894 cpu_manager.go:414] "containerMap: removing container" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134516 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134548 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" Oct 30 08:08:59.134666 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134555 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" Oct 30 08:08:59.134998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134844 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac068175-77c4-4983-ab26-1a95f4bbbb2a" containerName="revoke-keys" Oct 30 08:08:59.134998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134875 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b178415-02c9-444c-ae8c-8b907183c3c1" containerName="revoke-keys" Oct 30 08:08:59.134998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134888 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="225e48aa-8df4-4f7e-ad59-75448a449c6e" containerName="payload-pre-processing" Oct 30 08:08:59.134998 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.134906 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a27d8357-d617-4303-8e82-78ff0d6c38fd" containerName="payload-processing" Oct 30 08:08:59.142318 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.142281 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.158188 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.158155 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs"] Oct 30 08:08:59.163524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.163492 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:08:59.167302 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.167256 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h"] Oct 30 08:08:59.169158 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.169135 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:08:59.171350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.171331 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.178058 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.178031 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:08:59.178209 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.178116 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:08:59.191443 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.191413 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h"] Oct 30 08:08:59.207042 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.206998 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vff8\" (UniqueName: \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.207243 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.207133 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.207336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.207239 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.207336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.207326 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.207460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.207368 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.207544 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.207509 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsqwx\" (UniqueName: \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.308479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308425 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.308728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308484 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.308728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308517 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.308728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308600 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsqwx\" (UniqueName: \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.308728 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308690 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vff8\" (UniqueName: \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.308941 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.308760 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.309324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.309288 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.309324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.309308 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.311148 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.311128 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.311314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.311294 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.334962 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.334925 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsqwx\" (UniqueName: \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\") pod \"maas-api-revoke-sub-12ec7547-z27fs\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.334962 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.334946 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vff8\" (UniqueName: \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\") pod \"maas-api-revoke-sub-660990de-dsk2h\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.460522 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.460433 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:08:59.486143 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.486109 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:08:59.539230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.538557 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:08:59.771142 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:59.771110 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a2267d_ba44_4531_9e55_d4071e5283c7.slice/crio-ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e WatchSource:0}: Error finding container ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e: Status 404 returned error can't find the container with id ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e Oct 30 08:08:59.771685 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.771623 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs"] Oct 30 08:08:59.808990 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:08:59.808962 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7e5ea8_5295_47b8_93bd_1b5245f55110.slice/crio-094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e WatchSource:0}: Error finding container 094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e: Status 404 returned error can't find the container with id 094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e Oct 30 08:08:59.809098 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:08:59.809059 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h"] Oct 30 08:09:00.152254 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.152215 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:00.215780 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.215727 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" event={"ID":"34a2267d-ba44-4531-9e55-d4071e5283c7","Type":"ContainerStarted","Data":"af944615e16dc790f7f691775492abc54d7b31f62fbcf4d30e542546f7393077"} Oct 30 08:09:00.215994 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.215795 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" event={"ID":"34a2267d-ba44-4531-9e55-d4071e5283c7","Type":"ContainerStarted","Data":"ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e"} Oct 30 08:09:00.217512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.217481 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" event={"ID":"df7e5ea8-5295-47b8-93bd-1b5245f55110","Type":"ContainerStarted","Data":"ecf1d4c23ec7d6476a0054f646d50d97cf8022a3f79e6c214a1c9f9a39b28761"} Oct 30 08:09:00.217700 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.217536 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" event={"ID":"df7e5ea8-5295-47b8-93bd-1b5245f55110","Type":"ContainerStarted","Data":"094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e"} Oct 30 08:09:00.272475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.272405 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" podStartSLOduration=1.272390784 podStartE2EDuration="1.272390784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:00.270368837 +0000 UTC m=+4303.961286274" watchObservedRunningTime="2026-10-30 08:09:00.272390784 +0000 UTC m=+4303.963308220" Oct 30 08:09:00.274615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.274585 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:00.317842 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.317784 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" podStartSLOduration=1.317768867 podStartE2EDuration="1.317768867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:00.31653139 +0000 UTC m=+4304.007448827" watchObservedRunningTime="2026-10-30 08:09:00.317768867 +0000 UTC m=+4304.008686342" Oct 30 08:09:00.571452 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.571421 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:00.998575 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:00.998525 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:01.227805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.227724 5894 generic.go:356] "Generic (PLEG): container finished" podID="34a2267d-ba44-4531-9e55-d4071e5283c7" containerID="af944615e16dc790f7f691775492abc54d7b31f62fbcf4d30e542546f7393077" exitCode=0 Oct 30 08:09:01.227805 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.227782 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" event={"ID":"34a2267d-ba44-4531-9e55-d4071e5283c7","Type":"ContainerDied","Data":"af944615e16dc790f7f691775492abc54d7b31f62fbcf4d30e542546f7393077"} Oct 30 08:09:01.229670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.229642 5894 generic.go:356] "Generic (PLEG): container finished" podID="df7e5ea8-5295-47b8-93bd-1b5245f55110" containerID="ecf1d4c23ec7d6476a0054f646d50d97cf8022a3f79e6c214a1c9f9a39b28761" exitCode=0 Oct 30 08:09:01.229807 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.229726 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" event={"ID":"df7e5ea8-5295-47b8-93bd-1b5245f55110","Type":"ContainerDied","Data":"ecf1d4c23ec7d6476a0054f646d50d97cf8022a3f79e6c214a1c9f9a39b28761"} Oct 30 08:09:01.596212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.596174 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:01.728251 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.728212 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh"] Oct 30 08:09:01.732234 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.732200 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-4eff1f-vdsbh"] Oct 30 08:09:01.733423 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.733398 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:01.859961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.859871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:01.900650 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:01.900603 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:09:02.196111 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.196001 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:02.437955 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.437685 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:09:02.439672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.439618 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac068175-77c4-4983-ab26-1a95f4bbbb2a" path="/var/lib/kubelet/pods/ac068175-77c4-4983-ab26-1a95f4bbbb2a/volumes" Oct 30 08:09:02.441702 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.441684 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:09:02.549173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549141 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\") pod \"df7e5ea8-5295-47b8-93bd-1b5245f55110\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " Oct 30 08:09:02.549173 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549181 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\") pod \"34a2267d-ba44-4531-9e55-d4071e5283c7\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " Oct 30 08:09:02.549484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549253 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\") pod \"34a2267d-ba44-4531-9e55-d4071e5283c7\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " Oct 30 08:09:02.549484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549313 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\" (UniqueName: \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\") pod \"df7e5ea8-5295-47b8-93bd-1b5245f55110\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " Oct 30 08:09:02.549484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549401 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\" (UniqueName: \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\") pod \"34a2267d-ba44-4531-9e55-d4071e5283c7\" (UID: \"34a2267d-ba44-4531-9e55-d4071e5283c7\") " Oct 30 08:09:02.549484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549429 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\") pod \"df7e5ea8-5295-47b8-93bd-1b5245f55110\" (UID: \"df7e5ea8-5295-47b8-93bd-1b5245f55110\") " Oct 30 08:09:02.549814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.549784 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca" pod "34a2267d-ba44-4531-9e55-d4071e5283c7" (UID: "34a2267d-ba44-4531-9e55-d4071e5283c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:02.550052 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.550030 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:02.550255 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.550233 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca" pod "df7e5ea8-5295-47b8-93bd-1b5245f55110" (UID: "df7e5ea8-5295-47b8-93bd-1b5245f55110"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:02.552672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.552122 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca" pod "34a2267d-ba44-4531-9e55-d4071e5283c7" (UID: "34a2267d-ba44-4531-9e55-d4071e5283c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:02.552672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.552125 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca" pod "df7e5ea8-5295-47b8-93bd-1b5245f55110" (UID: "df7e5ea8-5295-47b8-93bd-1b5245f55110"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:02.552672 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.552296 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx" pod "34a2267d-ba44-4531-9e55-d4071e5283c7" (UID: "34a2267d-ba44-4531-9e55-d4071e5283c7"). InnerVolumeSpecName "kube-api-access-lsqwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:02.554434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.554408 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8" pod "df7e5ea8-5295-47b8-93bd-1b5245f55110" (UID: "df7e5ea8-5295-47b8-93bd-1b5245f55110"). InnerVolumeSpecName "kube-api-access-8vff8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:02.651330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.651295 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vff8\" (UniqueName: \"kubernetes.io/projected/df7e5ea8-5295-47b8-93bd-1b5245f55110-kube-api-access-8vff8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:02.651330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.651325 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsqwx\" (UniqueName: \"kubernetes.io/projected/34a2267d-ba44-4531-9e55-d4071e5283c7-kube-api-access-lsqwx\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:02.651330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.651336 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:02.651585 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.651345 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df7e5ea8-5295-47b8-93bd-1b5245f55110-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:02.651585 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:02.651354 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a2267d-ba44-4531-9e55-d4071e5283c7-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:03.246686 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.246653 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" Oct 30 08:09:03.246905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.246694 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12ec7547-z27fs" event={"ID":"34a2267d-ba44-4531-9e55-d4071e5283c7","Type":"ContainerDied","Data":"ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e"} Oct 30 08:09:03.246905 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.246778 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed96345d443602a45eb98b80efba5a4f2bdd6e63bb2eb6fe4015350ef3e9148e" Oct 30 08:09:03.248132 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.248111 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" Oct 30 08:09:03.248281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.248152 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660990de-dsk2h" event={"ID":"df7e5ea8-5295-47b8-93bd-1b5245f55110","Type":"ContainerDied","Data":"094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e"} Oct 30 08:09:03.248281 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.248197 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094d6ff31297649ec7c4a67a56aa129ee9f525dd9bc5c76c2330d440e62a6b9e" Oct 30 08:09:03.622900 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:03.622862 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:04.831315 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:04.831271 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:09:05.423393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:05.423354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:06.382388 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:06.382354 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:06.515903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:06.515864 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:07.245910 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:07.245871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:08.631542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:08.631495 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:08.748141 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:08.748096 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:08.977896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:08.977784 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:09.125766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.125730 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:09.484797 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.484759 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:09.687885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.687847 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp"] Oct 30 08:09:09.688437 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688422 5894 cpu_manager.go:414] "containerMap: removing container" podUID="34a2267d-ba44-4531-9e55-d4071e5283c7" containerName="revoke-keys" Oct 30 08:09:09.688489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688438 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a2267d-ba44-4531-9e55-d4071e5283c7" containerName="revoke-keys" Oct 30 08:09:09.688489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688450 5894 cpu_manager.go:414] "containerMap: removing container" podUID="df7e5ea8-5295-47b8-93bd-1b5245f55110" containerName="revoke-keys" Oct 30 08:09:09.688489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688458 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7e5ea8-5295-47b8-93bd-1b5245f55110" containerName="revoke-keys" Oct 30 08:09:09.688596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688586 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a2267d-ba44-4531-9e55-d4071e5283c7" containerName="revoke-keys" Oct 30 08:09:09.688675 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.688597 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7e5ea8-5295-47b8-93bd-1b5245f55110" containerName="revoke-keys" Oct 30 08:09:09.692580 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.692562 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.710901 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.710874 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp"] Oct 30 08:09:09.721515 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.721491 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:09:09.721658 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.721530 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:09:09.727682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.727655 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:09.727820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.727762 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:09.739275 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.739204 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.739462 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.739440 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.739539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.739491 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzlt9\" (UniqueName: \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.840506 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.840466 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.840772 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.840515 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzlt9\" (UniqueName: \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.840772 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.840573 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.841315 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.841292 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.843056 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.843031 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:09.866613 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:09.866581 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzlt9\" (UniqueName: \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\") pod \"maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:10.007291 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:10.007209 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:10.348986 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:09:10.348947 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62431d3_b125_4ec9_b28e_cfb0476d2f3c.slice/crio-bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67 WatchSource:0}: Error finding container bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67: Status 404 returned error can't find the container with id bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67 Oct 30 08:09:10.349906 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:10.349849 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp"] Oct 30 08:09:10.931842 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:10.931806 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:11.310321 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.310282 5894 generic.go:356] "Generic (PLEG): container finished" podID="e62431d3-b125-4ec9-b28e-cfb0476d2f3c" containerID="408273a88ce28cff0f7923fa78a577f9dd8aa2241dce58d0d30c3f29cbae92da" exitCode=0 Oct 30 08:09:11.310512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.310364 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" event={"ID":"e62431d3-b125-4ec9-b28e-cfb0476d2f3c","Type":"ContainerDied","Data":"408273a88ce28cff0f7923fa78a577f9dd8aa2241dce58d0d30c3f29cbae92da"} Oct 30 08:09:11.310512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.310418 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" event={"ID":"e62431d3-b125-4ec9-b28e-cfb0476d2f3c","Type":"ContainerStarted","Data":"bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67"} Oct 30 08:09:11.556930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.556891 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 08:09:11.714032 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.713992 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:11.841268 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:11.841180 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:12.448460 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.448421 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:12.509156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.509123 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:12.571846 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.571762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\") pod \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " Oct 30 08:09:12.572007 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.571963 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\" (UniqueName: \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\") pod \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " Oct 30 08:09:12.572079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.572015 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\") pod \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\" (UID: \"e62431d3-b125-4ec9-b28e-cfb0476d2f3c\") " Oct 30 08:09:12.572208 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.572180 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca" pod "e62431d3-b125-4ec9-b28e-cfb0476d2f3c" (UID: "e62431d3-b125-4ec9-b28e-cfb0476d2f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:12.572539 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.572515 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.574434 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.574401 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9" pod "e62431d3-b125-4ec9-b28e-cfb0476d2f3c" (UID: "e62431d3-b125-4ec9-b28e-cfb0476d2f3c"). InnerVolumeSpecName "kube-api-access-zzlt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:12.574902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.574878 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca" pod "e62431d3-b125-4ec9-b28e-cfb0476d2f3c" (UID: "e62431d3-b125-4ec9-b28e-cfb0476d2f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:12.673237 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.673195 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzlt9\" (UniqueName: \"kubernetes.io/projected/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-kube-api-access-zzlt9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.673237 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.673232 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e62431d3-b125-4ec9-b28e-cfb0476d2f3c-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:12.827114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.827019 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:12.951598 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:12.951562 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:13.071526 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.071451 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:13.326275 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.326242 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" Oct 30 08:09:13.326472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.326269 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp" event={"ID":"e62431d3-b125-4ec9-b28e-cfb0476d2f3c","Type":"ContainerDied","Data":"bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67"} Oct 30 08:09:13.326472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.326344 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff67e18c9023f489572cb1f221fe816941f9595cf7a80786840d27a8a1eec67" Oct 30 08:09:13.493350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.493310 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:13.632348 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.632249 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b"] Oct 30 08:09:13.633596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.633565 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:13.635130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.635108 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a0b3f5-knk7b"] Oct 30 08:09:13.751224 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:13.751183 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:14.122260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:14.122224 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:14.405902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:14.405812 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 08:09:14.417512 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:14.417477 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 08:09:14.434230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:14.434193 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b178415-02c9-444c-ae8c-8b907183c3c1" path="/var/lib/kubelet/pods/9b178415-02c9-444c-ae8c-8b907183c3c1/volumes" Oct 30 08:09:14.516215 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:14.516178 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:15.535623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:15.535588 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:16.557380 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:16.557342 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" containerID="cri-o://0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a" gracePeriod=25 Oct 30 08:09:17.227739 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:17.227710 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:19.033417 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:19.033384 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:19.187718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:19.187663 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:09:20.310659 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.310604 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:20.443464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.443428 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:20.506469 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.506429 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6"] Oct 30 08:09:20.507019 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.506999 5894 cpu_manager.go:414] "containerMap: removing container" podUID="e62431d3-b125-4ec9-b28e-cfb0476d2f3c" containerName="revoke-keys" Oct 30 08:09:20.507135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.507020 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62431d3-b125-4ec9-b28e-cfb0476d2f3c" containerName="revoke-keys" Oct 30 08:09:20.507225 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.507212 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62431d3-b125-4ec9-b28e-cfb0476d2f3c" containerName="revoke-keys" Oct 30 08:09:20.511668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.511626 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.565217 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.565132 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6"] Oct 30 08:09:20.575520 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.575489 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:20.575713 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.575549 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:20.631546 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.631509 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:09:20.631546 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.631509 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:09:20.667227 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.667192 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87f8\" (UniqueName: \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.667413 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.667380 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.667473 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.667413 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.768240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.768195 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.768240 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.768241 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.768552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.768315 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87f8\" (UniqueName: \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.769053 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.769024 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.770912 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.770891 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.794687 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.794657 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87f8\" (UniqueName: \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\") pod \"maas-api-revoke-sub-a2f85542-qcfp6\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.827160 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.826846 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:20.971168 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:20.971125 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:21.207077 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.207022 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6"] Oct 30 08:09:21.209521 ip-10-0-143-90 kubenswrapper[5894]: W1030 08:09:21.209497 5894 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfb35d2_4845_4920_8802_a3fdd00eeec1.slice/crio-5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952 WatchSource:0}: Error finding container 5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952: Status 404 returned error can't find the container with id 5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952 Oct 30 08:09:21.297156 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.297118 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:21.398325 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.398289 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" event={"ID":"0bfb35d2-4845-4920-8802-a3fdd00eeec1","Type":"ContainerStarted","Data":"5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952"} Oct 30 08:09:21.453397 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.453361 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:21.635067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.635020 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:09:21.635301 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.635274 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" containerID="cri-o://afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303" gracePeriod=30 Oct 30 08:09:21.670597 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:21.670551 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:22.087270 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.087238 5894 scope.go:122] "RemoveContainer" containerID="e637328affa888f951628404fd6f621a499e7bffc8e32b181f70402c4af27a31" Oct 30 08:09:22.102111 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.102075 5894 scope.go:122] "RemoveContainer" containerID="f93cbcd013989b411287dd8f8747f31f9b8202cd23f7de24b16fbd8b2bd81e8c" Oct 30 08:09:22.113526 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.113502 5894 scope.go:122] "RemoveContainer" containerID="d866978edde2346b2107ace3baa2d45d356e3bd9aad14946ec7b6b9de9a10764" Oct 30 08:09:22.123823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.123720 5894 scope.go:122] "RemoveContainer" containerID="0b73d5a3393e1a5239cdae10e06bf85df3dd76b6735b9076265762fef0cab0f0" Oct 30 08:09:22.156170 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.156142 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:22.395484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.395444 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh"] Oct 30 08:09:22.401363 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.401338 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.409375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.409337 5894 generic.go:356] "Generic (PLEG): container finished" podID="0bfb35d2-4845-4920-8802-a3fdd00eeec1" containerID="8767256ef1bf201b6d2b6ed415eb9bfae881cc9c7069447fee6e7353f6246672" exitCode=0 Oct 30 08:09:22.409532 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.409420 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" event={"ID":"0bfb35d2-4845-4920-8802-a3fdd00eeec1","Type":"ContainerDied","Data":"8767256ef1bf201b6d2b6ed415eb9bfae881cc9c7069447fee6e7353f6246672"} Oct 30 08:09:22.432774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.432738 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh"] Oct 30 08:09:22.504097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.503773 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.504097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.503986 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.504097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.504022 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9wg\" (UniqueName: \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.605593 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.605552 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.605593 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.605594 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9wg\" (UniqueName: \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.605885 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.605713 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.606260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.606237 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.608085 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.608064 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.635816 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.635786 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9wg\" (UniqueName: \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\") pod \"maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.717975 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.717889 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:22.719123 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.719088 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:22.838362 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.838307 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:22.960768 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:22.960728 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:23.032396 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.032358 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 08:09:23.032856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.032827 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" containerID="cri-o://23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d" gracePeriod=30 Oct 30 08:09:23.061668 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.059785 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh"] Oct 30 08:09:23.101141 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.101094 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:23.418880 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.418845 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" event={"ID":"6be4a3b2-52ef-4c5a-b99a-2230333530a1","Type":"ContainerStarted","Data":"529ed4c9c6f5dc1615ac347d43b64efb9883135a654aac1ebb0cd316877a431f"} Oct 30 08:09:23.586961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.586934 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:23.735731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.735694 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\") pod \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " Oct 30 08:09:23.735918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.735856 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\") pod \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " Oct 30 08:09:23.735918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.735907 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\" (UniqueName: \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\") pod \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\" (UID: \"0bfb35d2-4845-4920-8802-a3fdd00eeec1\") " Oct 30 08:09:23.736192 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.736157 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca" pod "0bfb35d2-4845-4920-8802-a3fdd00eeec1" (UID: "0bfb35d2-4845-4920-8802-a3fdd00eeec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:23.737983 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.737958 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca" pod "0bfb35d2-4845-4920-8802-a3fdd00eeec1" (UID: "0bfb35d2-4845-4920-8802-a3fdd00eeec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:23.738067 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.737966 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8" pod "0bfb35d2-4845-4920-8802-a3fdd00eeec1" (UID: "0bfb35d2-4845-4920-8802-a3fdd00eeec1"). InnerVolumeSpecName "kube-api-access-l87f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:23.838033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.837935 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.838033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.837979 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87f8\" (UniqueName: \"kubernetes.io/projected/0bfb35d2-4845-4920-8802-a3fdd00eeec1-kube-api-access-l87f8\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:23.838033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:23.837996 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bfb35d2-4845-4920-8802-a3fdd00eeec1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:24.426864 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.426814 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" Oct 30 08:09:24.429344 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.429305 5894 generic.go:356] "Generic (PLEG): container finished" podID="6be4a3b2-52ef-4c5a-b99a-2230333530a1" containerID="7bc9c001459e3d6b63e83404a25613bcf33ba1572639f39d70c5fd159065856d" exitCode=0 Oct 30 08:09:24.430545 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.430519 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2f85542-qcfp6" event={"ID":"0bfb35d2-4845-4920-8802-a3fdd00eeec1","Type":"ContainerDied","Data":"5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952"} Oct 30 08:09:24.430683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.430562 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aa56f4470ae285b136d459b4659905620d45a16de64039345f3f6700bbe6952" Oct 30 08:09:24.430683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.430571 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" event={"ID":"6be4a3b2-52ef-4c5a-b99a-2230333530a1","Type":"ContainerDied","Data":"7bc9c001459e3d6b63e83404a25613bcf33ba1572639f39d70c5fd159065856d"} Oct 30 08:09:24.768812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.768724 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:24.812954 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:24.812911 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:25.599750 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.599719 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:25.760087 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.760006 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\" (UniqueName: \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\") pod \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " Oct 30 08:09:25.760250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.760100 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\") pod \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " Oct 30 08:09:25.760320 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.760301 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\") pod \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\" (UID: \"6be4a3b2-52ef-4c5a-b99a-2230333530a1\") " Oct 30 08:09:25.760493 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.760469 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca" pod "6be4a3b2-52ef-4c5a-b99a-2230333530a1" (UID: "6be4a3b2-52ef-4c5a-b99a-2230333530a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:25.760761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.760741 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:25.762332 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.762296 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg" pod "6be4a3b2-52ef-4c5a-b99a-2230333530a1" (UID: "6be4a3b2-52ef-4c5a-b99a-2230333530a1"). InnerVolumeSpecName "kube-api-access-ln9wg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:25.762449 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.762396 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca" pod "6be4a3b2-52ef-4c5a-b99a-2230333530a1" (UID: "6be4a3b2-52ef-4c5a-b99a-2230333530a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:25.861679 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.861599 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6be4a3b2-52ef-4c5a-b99a-2230333530a1-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:25.861679 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:25.861626 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln9wg\" (UniqueName: \"kubernetes.io/projected/6be4a3b2-52ef-4c5a-b99a-2230333530a1-kube-api-access-ln9wg\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:26.449877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.449829 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" event={"ID":"6be4a3b2-52ef-4c5a-b99a-2230333530a1","Type":"ContainerDied","Data":"529ed4c9c6f5dc1615ac347d43b64efb9883135a654aac1ebb0cd316877a431f"} Oct 30 08:09:26.450063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.449909 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="529ed4c9c6f5dc1615ac347d43b64efb9883135a654aac1ebb0cd316877a431f" Oct 30 08:09:26.450063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.449990 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh" Oct 30 08:09:26.520801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.520762 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:26.655083 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.655048 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:26.789829 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.789796 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:26.912040 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:26.912009 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:09:27.048674 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.048619 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:27.083384 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083346 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.083623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083440 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.083623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083493 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.083623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083567 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.083623 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083609 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.084026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083714 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.084026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083751 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.084026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083845 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.084026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.083892 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\") pod \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\" (UID: \"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e\") " Oct 30 08:09:27.085447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.084389 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:27.085447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.084697 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:27.085447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.085186 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:27.085447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.085441 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:27.085872 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.085813 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:27.089952 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.089366 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "kube-api-access-7xrt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:27.091055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.091026 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:27.091619 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.091589 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:27.092468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.092443 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token" pod "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" (UID: "6c4e6f71-e06b-4fb4-9228-f35735f6bf2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:27.185847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185810 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.185847 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185844 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrt9\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-kube-api-access-7xrt9\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185860 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185870 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185879 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185887 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185896 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185903 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.186069 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.185911 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:27.252143 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.252087 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 08:09:27.263814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.263779 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 08:09:27.277177 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.277143 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 08:09:27.284489 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.284457 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:27.423571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.423471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:27.458552 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.458514 5894 generic.go:356] "Generic (PLEG): container finished" podID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerID="afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303" exitCode=0 Oct 30 08:09:27.458761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.458580 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" Oct 30 08:09:27.458761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.458598 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" event={"ID":"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e","Type":"ContainerDied","Data":"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303"} Oct 30 08:09:27.458761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.458708 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj" event={"ID":"6c4e6f71-e06b-4fb4-9228-f35735f6bf2e","Type":"ContainerDied","Data":"11d968aa2af55603c331c45db09b7f22519571564705e087534ba24dfc4f0265"} Oct 30 08:09:27.458761 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.458740 5894 scope.go:122] "RemoveContainer" containerID="afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303" Oct 30 08:09:27.473836 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.473805 5894 scope.go:122] "RemoveContainer" containerID="afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303" Oct 30 08:09:27.474218 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:27.474180 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303\": container with ID starting with afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303 not found: ID does not exist" containerID="afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303" Oct 30 08:09:27.474327 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.474224 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303"} err="failed to get container status \"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303\": rpc error: code = NotFound desc = could not find container \"afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303\": container with ID starting with afbf2aa8b7999953d39b8c0b92f4d16e4047cc0b3a5ca5c313eed5b6b7fbc303 not found: ID does not exist" Oct 30 08:09:27.521099 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.521067 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:09:27.537105 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:27.537072 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-bd38cf-openshift-default-7778676b8c-xjrfj"] Oct 30 08:09:28.099144 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.099106 5894 patch_prober.go:29] interesting pod/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 08:09:28.099614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.099166 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 08:09:28.219426 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.219396 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 08:09:28.297199 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297159 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297229 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297294 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297393 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297323 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297406 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297442 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297477 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.297579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297503 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:28.297579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297524 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.298078 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297809 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:28.298078 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.297979 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\") pod \"f044fbad-b5ea-4247-9a7b-7169b548759f\" (UID: \"f044fbad-b5ea-4247-9a7b-7169b548759f\") " Oct 30 08:09:28.298212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298115 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:28.298418 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298391 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:28.298570 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298554 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.298682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298575 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.298682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298589 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f044fbad-b5ea-4247-9a7b-7169b548759f-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.298682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298601 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.298871 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.298835 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:28.301708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.300985 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:28.301708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.301034 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "kube-api-access-l46wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:28.301708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.301040 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:28.301708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.301279 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo" pod "f044fbad-b5ea-4247-9a7b-7169b548759f" (UID: "f044fbad-b5ea-4247-9a7b-7169b548759f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:09:28.400046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.399999 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l46wk\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-kube-api-access-l46wk\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.400046 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.400036 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.400269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.400078 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f044fbad-b5ea-4247-9a7b-7169b548759f-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.400269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.400091 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.400269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.400102 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f044fbad-b5ea-4247-9a7b-7169b548759f-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:28.431014 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.430977 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" path="/var/lib/kubelet/pods/6c4e6f71-e06b-4fb4-9228-f35735f6bf2e/volumes" Oct 30 08:09:28.467315 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.467280 5894 generic.go:356] "Generic (PLEG): container finished" podID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerID="23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d" exitCode=0 Oct 30 08:09:28.467464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.467334 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" Oct 30 08:09:28.467464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.467364 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" event={"ID":"f044fbad-b5ea-4247-9a7b-7169b548759f","Type":"ContainerDied","Data":"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d"} Oct 30 08:09:28.467464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.467418 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x" event={"ID":"f044fbad-b5ea-4247-9a7b-7169b548759f","Type":"ContainerDied","Data":"e8ff5cf9533254205d46acdebb093087a1edb2b3c08b09e4f711d388f1a4dd13"} Oct 30 08:09:28.467464 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.467446 5894 scope.go:122] "RemoveContainer" containerID="23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d" Oct 30 08:09:28.477814 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.477795 5894 scope.go:122] "RemoveContainer" containerID="23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d" Oct 30 08:09:28.478099 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:28.478077 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d\": container with ID starting with 23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d not found: ID does not exist" containerID="23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d" Oct 30 08:09:28.478181 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.478107 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d"} err="failed to get container status \"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d\": rpc error: code = NotFound desc = could not find container \"23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d\": container with ID starting with 23835e29d50d0030cf37ac61f39bc507718879578fe0af51e311f89b52b2b74d not found: ID does not exist" Oct 30 08:09:28.530057 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.530028 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 08:09:28.535659 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.535612 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a0b3f5-openshift-default-bb859966f-lmr6x"] Oct 30 08:09:28.656589 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.656492 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:28.852088 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:28.852045 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:29.429708 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.429667 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" containerID="cri-o://070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9" gracePeriod=45 Oct 30 08:09:29.443117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.443079 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" containerID="cri-o://3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530" gracePeriod=45 Oct 30 08:09:29.722857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.722826 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 08:09:29.742895 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.742867 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 08:09:29.815022 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.814977 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815035 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\" (UniqueName: \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815079 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815206 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815173 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\" (UniqueName: \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815241 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815279 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815311 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815297 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815358 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815369 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.815412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815408 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\") pod \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\" (UID: \"9fa6b8eb-4082-4703-aa04-7a3d8165a505\") " Oct 30 08:09:29.815917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815479 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815502 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815595 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\") pod \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\" (UID: \"6ebe0fb1-3f63-4b93-b602-833f29138e5c\") " Oct 30 08:09:29.815917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.815866 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.816120 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816052 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.816120 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816070 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.816120 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816087 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.816315 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816292 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.816722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816610 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.816722 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.816695 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.818189 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.818148 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "kube-api-access-vlqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:29.818780 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.818728 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:29.818780 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.818746 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "kube-api-access-2jvpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:29.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.820697 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.820721 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs" pod "9fa6b8eb-4082-4703-aa04-7a3d8165a505" (UID: "9fa6b8eb-4082-4703-aa04-7a3d8165a505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:29.823670 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.822698 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm" pod "6ebe0fb1-3f63-4b93-b602-833f29138e5c" (UID: "6ebe0fb1-3f63-4b93-b602-833f29138e5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:29.917445 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917416 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlqj6\" (UniqueName: \"kubernetes.io/projected/6ebe0fb1-3f63-4b93-b602-833f29138e5c-kube-api-access-vlqj6\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917445 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917442 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917445 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917454 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fa6b8eb-4082-4703-aa04-7a3d8165a505-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917463 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917470 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9fa6b8eb-4082-4703-aa04-7a3d8165a505-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917478 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917485 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917491 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ebe0fb1-3f63-4b93-b602-833f29138e5c-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:29.917699 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:29.917499 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvpc\" (UniqueName: \"kubernetes.io/projected/9fa6b8eb-4082-4703-aa04-7a3d8165a505-kube-api-access-2jvpc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:30.425849 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.425812 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:30.430820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.430791 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" path="/var/lib/kubelet/pods/f044fbad-b5ea-4247-9a7b-7169b548759f/volumes" Oct 30 08:09:30.485235 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.485193 5894 generic.go:356] "Generic (PLEG): container finished" podID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerID="070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9" exitCode=0 Oct 30 08:09:30.485411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.485246 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" Oct 30 08:09:30.485411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.485280 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" event={"ID":"6ebe0fb1-3f63-4b93-b602-833f29138e5c","Type":"ContainerDied","Data":"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9"} Oct 30 08:09:30.485411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.485321 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68" event={"ID":"6ebe0fb1-3f63-4b93-b602-833f29138e5c","Type":"ContainerDied","Data":"9d762eaffe6b62886ca08eec254d466f94333b31f73ae542b0cf346d7b625fde"} Oct 30 08:09:30.485411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.485338 5894 scope.go:122] "RemoveContainer" containerID="070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9" Oct 30 08:09:30.486977 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.486950 5894 generic.go:356] "Generic (PLEG): container finished" podID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerID="3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530" exitCode=0 Oct 30 08:09:30.487062 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.486989 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" Oct 30 08:09:30.487062 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.487008 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" event={"ID":"9fa6b8eb-4082-4703-aa04-7a3d8165a505","Type":"ContainerDied","Data":"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530"} Oct 30 08:09:30.487062 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.487052 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg" event={"ID":"9fa6b8eb-4082-4703-aa04-7a3d8165a505","Type":"ContainerDied","Data":"762bc8bb4f459e0b8bad6dcb84e358739147338f81dcd69679c96291d74e3216"} Oct 30 08:09:30.495845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.495818 5894 scope.go:122] "RemoveContainer" containerID="070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9" Oct 30 08:09:30.496165 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:30.496137 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9\": container with ID starting with 070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9 not found: ID does not exist" containerID="070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9" Oct 30 08:09:30.496250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.496179 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9"} err="failed to get container status \"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9\": rpc error: code = NotFound desc = could not find container \"070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9\": container with ID starting with 070ba9034aeabcdf971f81d6549ef44f1df5dec8e9d2ff40da652831dccfc8d9 not found: ID does not exist" Oct 30 08:09:30.496250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.496198 5894 scope.go:122] "RemoveContainer" containerID="3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530" Oct 30 08:09:30.506379 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.506358 5894 scope.go:122] "RemoveContainer" containerID="3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530" Oct 30 08:09:30.506683 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:30.506655 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530\": container with ID starting with 3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530 not found: ID does not exist" containerID="3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530" Oct 30 08:09:30.506786 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.506692 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530"} err="failed to get container status \"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530\": rpc error: code = NotFound desc = could not find container \"3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530\": container with ID starting with 3c3cbd7f7cc9d37468bfd10b710791f77134ae428b7c8e6ef9485e174f093530 not found: ID does not exist" Oct 30 08:09:30.548484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.548443 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 08:09:30.562513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.562473 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/distinct-2-w2-e5fa2d-kserve-bf54b6c7c-scq68"] Oct 30 08:09:30.595079 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.595044 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 08:09:30.599411 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:30.599369 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e5fa2d/premium-w2-e5fa2d-kserve-546fd4f85f-x7gqg"] Oct 30 08:09:32.229774 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:32.229732 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:32.431026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:32.430991 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" path="/var/lib/kubelet/pods/6ebe0fb1-3f63-4b93-b602-833f29138e5c/volumes" Oct 30 08:09:32.431501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:32.431484 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" path="/var/lib/kubelet/pods/9fa6b8eb-4082-4703-aa04-7a3d8165a505/volumes" Oct 30 08:09:32.976682 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:32.976648 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:33.099088 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:33.099054 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:33.638844 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:33.638804 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:34.007651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.007478 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc"] Oct 30 08:09:34.008802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008765 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" Oct 30 08:09:34.008802 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008796 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008835 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008843 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008858 5894 cpu_manager.go:414] "containerMap: removing container" podUID="6be4a3b2-52ef-4c5a-b99a-2230333530a1" containerName="revoke-keys" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008866 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be4a3b2-52ef-4c5a-b99a-2230333530a1" containerName="revoke-keys" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008882 5894 cpu_manager.go:414] "containerMap: removing container" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008890 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008913 5894 cpu_manager.go:414] "containerMap: removing container" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008920 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008930 5894 cpu_manager.go:414] "containerMap: removing container" podUID="0bfb35d2-4845-4920-8802-a3fdd00eeec1" containerName="revoke-keys" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.008938 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfb35d2-4845-4920-8802-a3fdd00eeec1" containerName="revoke-keys" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009056 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="f044fbad-b5ea-4247-9a7b-7169b548759f" containerName="istio-proxy" Oct 30 08:09:34.009063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009068 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfb35d2-4845-4920-8802-a3fdd00eeec1" containerName="revoke-keys" Oct 30 08:09:34.009863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009090 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c4e6f71-e06b-4fb4-9228-f35735f6bf2e" containerName="istio-proxy" Oct 30 08:09:34.009863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009106 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebe0fb1-3f63-4b93-b602-833f29138e5c" containerName="main" Oct 30 08:09:34.009863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009116 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be4a3b2-52ef-4c5a-b99a-2230333530a1" containerName="revoke-keys" Oct 30 08:09:34.009863 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.009124 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa6b8eb-4082-4703-aa04-7a3d8165a505" containerName="main" Oct 30 08:09:34.014829 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.014800 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.027801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.027773 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc"] Oct 30 08:09:34.038558 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.038529 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 08:09:34.038742 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.038601 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 08:09:34.064205 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.064167 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lcc\" (UniqueName: \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.064395 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.064292 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.064395 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.064392 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.110765 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.110733 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 08:09:34.110966 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.110917 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mq4nv\"" Oct 30 08:09:34.138073 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.138033 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:34.166058 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.166017 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.166253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.166108 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.166253 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.166237 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lcc\" (UniqueName: \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.166803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.166781 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.168472 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.168446 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.192459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.192428 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lcc\" (UniqueName: \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\") pod \"maas-api-revoke-sub-425f29b9-4xfpc\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.263566 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.263471 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:34.303902 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.303865 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:09:34.333346 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.332568 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:34.343692 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.343593 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp"] Oct 30 08:09:34.347456 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.347419 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e5fa2d-95jwp"] Oct 30 08:09:34.387390 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.387357 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:34.432233 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.432196 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62431d3-b125-4ec9-b28e-cfb0476d2f3c" path="/var/lib/kubelet/pods/e62431d3-b125-4ec9-b28e-cfb0476d2f3c/volumes" Oct 30 08:09:34.650528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.650376 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc"] Oct 30 08:09:34.769317 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.769280 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 08:09:34.833475 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:34.833435 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:09:34.846995 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:34.846948 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:35.526266 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:35.526184 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" event={"ID":"a55febc7-7879-47b5-892a-766afc714734","Type":"ContainerStarted","Data":"8fd6836386c695357daa96cb6165c09e6cfdd5a6f460420c537f3cbe3ce33a31"} Oct 30 08:09:35.526266 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:35.526240 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" event={"ID":"a55febc7-7879-47b5-892a-766afc714734","Type":"ContainerStarted","Data":"ba68fa3fbbbeb15a876d1d5f4ee286bc9aaa9ec6fe5ccde5dc24930925577b83"} Oct 30 08:09:35.573840 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:35.573786 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" podStartSLOduration=2.573771242 podStartE2EDuration="2.573771242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 08:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:09:35.57120486 +0000 UTC m=+4339.262122297" watchObservedRunningTime="2026-10-30 08:09:35.573771242 +0000 UTC m=+4339.264688678" Oct 30 08:09:35.639135 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:35.639103 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:36.534982 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.534948 5894 generic.go:356] "Generic (PLEG): container finished" podID="a55febc7-7879-47b5-892a-766afc714734" containerID="8fd6836386c695357daa96cb6165c09e6cfdd5a6f460420c537f3cbe3ce33a31" exitCode=0 Oct 30 08:09:36.535366 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.535039 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" event={"ID":"a55febc7-7879-47b5-892a-766afc714734","Type":"ContainerDied","Data":"8fd6836386c695357daa96cb6165c09e6cfdd5a6f460420c537f3cbe3ce33a31"} Oct 30 08:09:36.827801 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.827774 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 08:09:36.896524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.896482 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\") pod \"3c913759-de4e-40c6-8169-d3f511319963\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " Oct 30 08:09:36.896756 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.896545 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\" (UniqueName: \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\") pod \"3c913759-de4e-40c6-8169-d3f511319963\" (UID: \"3c913759-de4e-40c6-8169-d3f511319963\") " Oct 30 08:09:36.896953 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.896929 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume" pod "3c913759-de4e-40c6-8169-d3f511319963" (UID: "3c913759-de4e-40c6-8169-d3f511319963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:36.897041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.897021 5894 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c913759-de4e-40c6-8169-d3f511319963-plugins-config-volume\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:36.898730 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.898707 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt" pod "3c913759-de4e-40c6-8169-d3f511319963" (UID: "3c913759-de4e-40c6-8169-d3f511319963"). InnerVolumeSpecName "kube-api-access-bspxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:36.998357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:36.998322 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bspxt\" (UniqueName: \"kubernetes.io/projected/3c913759-de4e-40c6-8169-d3f511319963-kube-api-access-bspxt\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:37.543524 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.543490 5894 generic.go:356] "Generic (PLEG): container finished" podID="3c913759-de4e-40c6-8169-d3f511319963" containerID="0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a" exitCode=0 Oct 30 08:09:37.544101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.543554 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" Oct 30 08:09:37.544101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.543577 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" event={"ID":"3c913759-de4e-40c6-8169-d3f511319963","Type":"ContainerDied","Data":"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a"} Oct 30 08:09:37.544101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.543652 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh" event={"ID":"3c913759-de4e-40c6-8169-d3f511319963","Type":"ContainerDied","Data":"81f2c7ab9151b985695cae8d75587bbe730492e6f0b10af9b868963153fdb02e"} Oct 30 08:09:37.544101 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.543676 5894 scope.go:122] "RemoveContainer" containerID="0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a" Oct 30 08:09:37.554789 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.554760 5894 scope.go:122] "RemoveContainer" containerID="0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a" Oct 30 08:09:37.555123 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:37.555100 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a\": container with ID starting with 0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a not found: ID does not exist" containerID="0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a" Oct 30 08:09:37.555212 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.555136 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a"} err="failed to get container status \"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a\": rpc error: code = NotFound desc = could not find container \"0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a\": container with ID starting with 0355bcba023bbc8a91b3b8f85b49e54ffbc92b08105f3669cded7158eff4d14a not found: ID does not exist" Oct 30 08:09:37.609825 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.605104 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 08:09:37.620373 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.620328 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-bd38cf-6dc57d8d8-mzvsh"] Oct 30 08:09:37.694272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.694248 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:37.808507 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.808424 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\") pod \"a55febc7-7879-47b5-892a-766afc714734\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " Oct 30 08:09:37.808688 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.808623 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\") pod \"a55febc7-7879-47b5-892a-766afc714734\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " Oct 30 08:09:37.808745 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.808685 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\" (UniqueName: \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\") pod \"a55febc7-7879-47b5-892a-766afc714734\" (UID: \"a55febc7-7879-47b5-892a-766afc714734\") " Oct 30 08:09:37.809047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.809012 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca" pod "a55febc7-7879-47b5-892a-766afc714734" (UID: "a55febc7-7879-47b5-892a-766afc714734"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:09:37.810562 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.810543 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca" pod "a55febc7-7879-47b5-892a-766afc714734" (UID: "a55febc7-7879-47b5-892a-766afc714734"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:37.810853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.810834 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc" pod "a55febc7-7879-47b5-892a-766afc714734" (UID: "a55febc7-7879-47b5-892a-766afc714734"). InnerVolumeSpecName "kube-api-access-v8lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:37.910442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.910405 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55febc7-7879-47b5-892a-766afc714734-maas-api-xks-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:37.910442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.910437 5894 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55febc7-7879-47b5-892a-766afc714734-maas-api-openshift-ca\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:37.910442 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:37.910449 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8lcc\" (UniqueName: \"kubernetes.io/projected/a55febc7-7879-47b5-892a-766afc714734-kube-api-access-v8lcc\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:38.430474 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:38.430438 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c913759-de4e-40c6-8169-d3f511319963" path="/var/lib/kubelet/pods/3c913759-de4e-40c6-8169-d3f511319963/volumes" Oct 30 08:09:38.551964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:38.551925 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" Oct 30 08:09:38.551964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:38.551951 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425f29b9-4xfpc" event={"ID":"a55febc7-7879-47b5-892a-766afc714734","Type":"ContainerDied","Data":"ba68fa3fbbbeb15a876d1d5f4ee286bc9aaa9ec6fe5ccde5dc24930925577b83"} Oct 30 08:09:38.552479 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:38.552005 5894 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba68fa3fbbbeb15a876d1d5f4ee286bc9aaa9ec6fe5ccde5dc24930925577b83" Oct 30 08:09:41.033047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:41.033008 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq"] Oct 30 08:09:41.047469 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:41.047438 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c08342fb-nq8gq"] Oct 30 08:09:41.049701 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:41.049676 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m"] Oct 30 08:09:41.051548 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:41.051514 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c5213f7-6z29m"] Oct 30 08:09:42.116876 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.116842 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:42.246501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.246463 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:42.274553 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.274513 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" containerID="cri-o://de5858fd3890cac142125c2f2adc61f0bf7916693770ebff209acb0f7af7c6fc" gracePeriod=45 Oct 30 08:09:42.285475 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.285433 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" containerID="cri-o://c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef" gracePeriod=45 Oct 30 08:09:42.303660 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.303600 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" containerID="cri-o://ad9b7c6904a999bacaf63df19e785f2502e64852b7d62e08383606dc5a59ea1c" gracePeriod=45 Oct 30 08:09:42.362470 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.362437 5894 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": dial tcp 10.134.0.47:8000: connect: connection refused" Oct 30 08:09:42.439581 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.438624 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32e3508c-824e-4326-bb13-8d3b598d5ff9" path="/var/lib/kubelet/pods/32e3508c-824e-4326-bb13-8d3b598d5ff9/volumes" Oct 30 08:09:42.439581 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.439349 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92721e74-7b49-4af6-9e98-91a8c2b4a374" path="/var/lib/kubelet/pods/92721e74-7b49-4af6-9e98-91a8c2b4a374/volumes" Oct 30 08:09:42.440936 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:09:42.440863 5894 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe76c9f_e2ef_471b_a29b_5d9caee9947a.slice/crio-conmon-c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef.scope\": RecentStats: unable to find data in memory cache]" Oct 30 08:09:42.461314 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.461161 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:42.611711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.608531 5894 generic.go:356] "Generic (PLEG): container finished" podID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerID="c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef" exitCode=0 Oct 30 08:09:42.611711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.608649 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" event={"ID":"fbe76c9f-e2ef-471b-a29b-5d9caee9947a","Type":"ContainerDied","Data":"c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef"} Oct 30 08:09:42.611711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.610388 5894 generic.go:356] "Generic (PLEG): container finished" podID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerID="ad9b7c6904a999bacaf63df19e785f2502e64852b7d62e08383606dc5a59ea1c" exitCode=0 Oct 30 08:09:42.611711 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.610551 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" event={"ID":"973f8c38-94b3-4869-8df0-2691c6ecb509","Type":"ContainerDied","Data":"ad9b7c6904a999bacaf63df19e785f2502e64852b7d62e08383606dc5a59ea1c"} Oct 30 08:09:42.613651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.612234 5894 generic.go:356] "Generic (PLEG): container finished" podID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerID="de5858fd3890cac142125c2f2adc61f0bf7916693770ebff209acb0f7af7c6fc" exitCode=0 Oct 30 08:09:42.613651 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.612303 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" event={"ID":"c517bb18-ee3d-4b4c-a49d-911b92922de7","Type":"ContainerDied","Data":"de5858fd3890cac142125c2f2adc61f0bf7916693770ebff209acb0f7af7c6fc"} Oct 30 08:09:42.711836 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.709805 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 08:09:42.736768 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.736741 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 08:09:42.745537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.745503 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 08:09:42.764655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764611 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.764856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764727 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.764856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764762 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.764856 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764825 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.765090 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764878 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\" (UniqueName: \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.765090 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764917 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\") pod \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\" (UID: \"fbe76c9f-e2ef-471b-a29b-5d9caee9947a\") " Oct 30 08:09:42.766350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.764910 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.766350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.765081 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.766350 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.765328 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.776551 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.776513 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.777759 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.777717 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "kube-api-access-dk5r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:42.778166 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.778140 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs" pod "fbe76c9f-e2ef-471b-a29b-5d9caee9947a" (UID: "fbe76c9f-e2ef-471b-a29b-5d9caee9947a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:42.866272 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866233 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866290 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866344 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866379 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866414 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.866458 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866446 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866491 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\" (UniqueName: \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866519 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\" (UniqueName: \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866569 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\" (UniqueName: \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\") pod \"c517bb18-ee3d-4b4c-a49d-911b92922de7\" (UID: \"c517bb18-ee3d-4b4c-a49d-911b92922de7\") " Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866652 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866672 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.866712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866702 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.867063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866773 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\" (UniqueName: \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\") pod \"973f8c38-94b3-4869-8df0-2691c6ecb509\" (UID: \"973f8c38-94b3-4869-8df0-2691c6ecb509\") " Oct 30 08:09:42.867063 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.866981 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.867307 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867287 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867310 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867323 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867355 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867368 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867397 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk5r5\" (UniqueName: \"kubernetes.io/projected/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-kube-api-access-dk5r5\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867428 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867486 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe76c9f-e2ef-471b-a29b-5d9caee9947a-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.867843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867514 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.867843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867723 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.867843 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.867772 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.868269 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.868026 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.880064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.879989 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.880064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.880021 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "kube-api-access-4w425". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:42.880064 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.880039 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "kube-api-access-gtmgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:09:42.880485 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.880464 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs" pod "973f8c38-94b3-4869-8df0-2691c6ecb509" (UID: "973f8c38-94b3-4869-8df0-2691c6ecb509"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:42.881967 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.881904 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:09:42.881967 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.881920 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs" pod "c517bb18-ee3d-4b4c-a49d-911b92922de7" (UID: "c517bb18-ee3d-4b4c-a49d-911b92922de7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.968973 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/973f8c38-94b3-4869-8df0-2691c6ecb509-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969007 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969017 5894 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-tmp-dir\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969025 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c517bb18-ee3d-4b4c-a49d-911b92922de7-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969033 5894 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-model-cache\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969040 5894 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c517bb18-ee3d-4b4c-a49d-911b92922de7-tls-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969048 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w425\" (UniqueName: \"kubernetes.io/projected/973f8c38-94b3-4869-8df0-2691c6ecb509-kube-api-access-4w425\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969057 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmgm\" (UniqueName: \"kubernetes.io/projected/c517bb18-ee3d-4b4c-a49d-911b92922de7-kube-api-access-gtmgm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969068 5894 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-home\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:42.969459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:42.969076 5894 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/973f8c38-94b3-4869-8df0-2691c6ecb509-dshm\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:09:43.620693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.620588 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" Oct 30 08:09:43.620693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.620603 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh" event={"ID":"fbe76c9f-e2ef-471b-a29b-5d9caee9947a","Type":"ContainerDied","Data":"bd6c38cf04d4b680cf2797518fa54e274d244fb08cec4a19ded462f5931f06bc"} Oct 30 08:09:43.620693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.620685 5894 scope.go:122] "RemoveContainer" containerID="c51a1c9d5fc3e7f85514fb7bae9ae26f76421b63ca6ecbdd9b8c6da5d632fcef" Oct 30 08:09:43.622596 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.622577 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" Oct 30 08:09:43.622737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.622627 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5" event={"ID":"973f8c38-94b3-4869-8df0-2691c6ecb509","Type":"ContainerDied","Data":"10516d74d1471281d5052d25d661c23de0fc2accd1d4d117ecf8d21a8b39a0ec"} Oct 30 08:09:43.624517 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.624498 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" Oct 30 08:09:43.624655 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.624531 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk" event={"ID":"c517bb18-ee3d-4b4c-a49d-911b92922de7","Type":"ContainerDied","Data":"7e70ba07c0eadffade05350233b9b1bf0b64017ddb19f4f324fabcfb641c7753"} Oct 30 08:09:43.644164 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.644139 5894 scope.go:122] "RemoveContainer" containerID="ad9b7c6904a999bacaf63df19e785f2502e64852b7d62e08383606dc5a59ea1c" Oct 30 08:09:43.653844 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.653818 5894 scope.go:122] "RemoveContainer" containerID="de5858fd3890cac142125c2f2adc61f0bf7916693770ebff209acb0f7af7c6fc" Oct 30 08:09:43.699848 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.699777 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 08:09:43.714231 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.714198 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-w3-2f845b-kserve-79f848d597-ffrqh"] Oct 30 08:09:43.746997 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.746959 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 08:09:43.750850 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.750812 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/premium-w3-2f845b-kserve-676d96cc4-tv8t5"] Oct 30 08:09:43.788978 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.788945 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 08:09:43.795574 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:43.795541 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2f845b/distinct-2-w3-2f845b-kserve-66f7fdd586-w2wlk"] Oct 30 08:09:44.429766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:44.429734 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" path="/var/lib/kubelet/pods/973f8c38-94b3-4869-8df0-2691c6ecb509/volumes" Oct 30 08:09:44.430179 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:44.430165 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" path="/var/lib/kubelet/pods/c517bb18-ee3d-4b4c-a49d-911b92922de7/volumes" Oct 30 08:09:44.430547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:44.430535 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" path="/var/lib/kubelet/pods/fbe76c9f-e2ef-471b-a29b-5d9caee9947a/volumes" Oct 30 08:09:45.280884 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:45.280825 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:45.399609 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:45.399575 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:46.021845 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:46.021805 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:46.496043 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:46.496004 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:46.622710 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:46.622520 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:46.630259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:46.630225 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh"] Oct 30 08:09:46.644336 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:46.644304 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2f845b-ncxsh"] Oct 30 08:09:47.146783 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:47.146751 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:48.208878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:48.208839 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-wg9tq"] Oct 30 08:09:48.430330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:48.430294 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be4a3b2-52ef-4c5a-b99a-2230333530a1" path="/var/lib/kubelet/pods/6be4a3b2-52ef-4c5a-b99a-2230333530a1/volumes" Oct 30 08:09:50.834967 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:50.834921 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:50.955026 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:50.954988 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:53.383294 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:53.383258 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:54.360267 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:54.360240 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:54.489476 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:54.489421 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:54.722505 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:54.722407 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:56.250261 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:56.250226 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5c2cx_01b69050-e4b3-483e-8065-0ef33a849871/manager/0.log" Oct 30 08:09:56.438917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:56.438882 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-p6tlb_4864a195-5b08-4c9b-b91c-5312624dc72e/kuadrant-console-plugin/0.log" Oct 30 08:09:57.101615 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:57.101584 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wg9tq_13d4950f-8dc2-43d5-8969-842bc081873c/limitador/0.log" Oct 30 08:09:58.304207 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:58.304177 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:09:58.507586 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:58.507554 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:09:58.729288 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:58.729204 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:09:59.308556 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:59.308511 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:09:59.523426 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:59.523393 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:09:59.660322 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:59.660234 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:09:59.727540 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:59.727502 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:09:59.974890 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:09:59.974814 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:00.389614 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:00.389579 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:04.205114 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:04.205078 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:04.321731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:04.321693 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:04.830145 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:10:04.830100 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:10:05.812914 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:05.812871 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:11.499446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:11.499397 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:11.895903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:11.895869 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:12.095509 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:12.095472 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:12.318016 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:12.317985 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:13.055330 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:13.055294 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:13.239433 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:13.239409 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:13.448304 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:13.448220 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:13.688564 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:13.688533 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:14.080300 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:14.080265 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:14.880945 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:14.880905 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-12ec7547-z27fs_34a2267d-ba44-4531-9e55-d4071e5283c7/revoke-keys/0.log" Oct 30 08:10:15.072937 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:15.072910 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2874b07e-p5vjs_8a5eb796-706c-4cbb-b3a3-8a91c14fa315/revoke-keys/0.log" Oct 30 08:10:15.266790 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:15.266692 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-318d39cf-bznnq_e21f34cf-b472-4fad-a8c4-b2ee6af46079/revoke-keys/0.log" Oct 30 08:10:15.449167 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:15.449133 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-425f29b9-4xfpc_a55febc7-7879-47b5-892a-766afc714734/revoke-keys/0.log" Oct 30 08:10:15.640951 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:15.640908 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-4c66bf60-v288v_3050a307-8385-4ecf-b84a-20f6396d8afb/revoke-keys/0.log" Oct 30 08:10:15.825896 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:15.825853 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-582beb14-l8vlf_932c3733-e64e-4980-ac4a-c87bb7ab593a/revoke-keys/0.log" Oct 30 08:10:16.006961 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:16.006878 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-660990de-dsk2h_df7e5ea8-5295-47b8-93bd-1b5245f55110/revoke-keys/0.log" Oct 30 08:10:16.401033 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:16.401003 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6ea667b3-r82zr_9e52df3d-9eb3-41f5-acdb-42670fe8791e/revoke-keys/0.log" Oct 30 08:10:16.590027 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:16.589997 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7f55da64-2kdrd_5353cb5c-8121-4ba8-9db7-a82249758e41/revoke-keys/0.log" Oct 30 08:10:16.981707 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:16.981675 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a2f85542-qcfp6_0bfb35d2-4845-4920-8802-a3fdd00eeec1/revoke-keys/0.log" Oct 30 08:10:17.177412 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:17.177383 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ab551f54-hlxvr_36c6c883-8456-403b-909c-e79a3425d19d/revoke-keys/0.log" Oct 30 08:10:17.373047 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:17.373019 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-c0ebcc12-qrz2q_e2179221-3889-471b-929e-0fac6409a40b/revoke-keys/0.log" Oct 30 08:10:17.557689 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:17.557657 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e67fd566-t46nl_53b9a27a-647a-4098-94ca-7372355d9282/revoke-keys/0.log" Oct 30 08:10:17.753964 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:17.753878 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fe25ed43-nl248_3da6bcdc-e7ae-4465-baa2-3de77d5b1b47/revoke-keys/0.log" Oct 30 08:10:19.947218 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:19.947171 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5c2cx_01b69050-e4b3-483e-8065-0ef33a849871/manager/0.log" Oct 30 08:10:20.132232 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:20.132199 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-p6tlb_4864a195-5b08-4c9b-b91c-5312624dc72e/kuadrant-console-plugin/0.log" Oct 30 08:10:20.677176 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:20.677148 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wg9tq_13d4950f-8dc2-43d5-8969-842bc081873c/limitador/0.log" Oct 30 08:10:21.817169 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:21.817134 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nf7b5_e60bb623-53ca-42c5-af7d-610ee177be92/discovery/0.log" Oct 30 08:10:22.012003 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.011976 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-5mvd9_35207e99-083b-4214-bf26-79fabec5c2bf/kube-auth-proxy/0.log" Oct 30 08:10:22.177326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.177242 5894 scope.go:122] "RemoveContainer" containerID="cfcc32de6a21acd8ed865294d2bcc9241c69d448b5620e9767aaaa6ac142a5fb" Oct 30 08:10:22.187820 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.187797 5894 scope.go:122] "RemoveContainer" containerID="962eaa56b5cb6bd4fcf6dd4452a16f03744abddb0e160b123a37424a9972dea0" Oct 30 08:10:22.197823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.197798 5894 scope.go:122] "RemoveContainer" containerID="1af9582afb4c5ce00b42a1ff91ec9f94d84ed5ec0c814923a6c6022f259924f4" Oct 30 08:10:22.208011 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.207983 5894 scope.go:122] "RemoveContainer" containerID="cbb117e94cc6c9bfa22ce29af09e2bbe3f8f6318596550d9dbe6e7bec218b668" Oct 30 08:10:22.219022 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.218994 5894 scope.go:122] "RemoveContainer" containerID="3c3dfe0e66125c8c9e25920454dfa8926d84dc7ba7af130c758a5408c91fcdb5" Oct 30 08:10:22.228053 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.228025 5894 scope.go:122] "RemoveContainer" containerID="99c70fc11a000453d6a4c7237b267c01ebd202e90ee5df3efed486cf78adddc4" Oct 30 08:10:22.597696 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:22.597660 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-767jp_54d37e1d-1a39-4f9c-8f4f-9163bd1197a3/istio-proxy/0.log" Oct 30 08:10:23.008579 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:23.008498 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vpdnn_cd6421ad-3f96-4128-9aae-9bd4ba795a75/istio-proxy/0.log" Oct 30 08:10:34.339921 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:34.339852 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:34.827318 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:10:34.827278 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:10:36.351275 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:36.351239 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:36.510009 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:36.509966 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:37.133438 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:37.133403 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 08:10:37.133683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:37.133585 5894 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" containerID="cri-o://59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15" gracePeriod=30 Oct 30 08:10:41.159284 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:41.159247 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5n2mb_6ca9fe1e-35ee-469e-b4af-7c13480101f7/global-pull-secret-syncer/0.log" Oct 30 08:10:42.215447 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.215413 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" Oct 30 08:10:42.402726 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.402698 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 08:10:42.479788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.479702 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.479788 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.479782 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.479920 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.479978 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480055 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480014 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480077 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480082 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:42.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480157 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480225 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480347 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480222 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:42.480620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480258 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:42.480620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480116 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 08:10:42.480620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480487 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:42.480620 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480569 5894 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\") pod \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\" (UID: \"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3\") " Oct 30 08:10:42.480924 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.480902 5894 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-data\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.481110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.481077 5894 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istiod-ca-cert\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.481110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.481102 5894 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.481293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.481116 5894 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-workload-certs\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.481293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.481132 5894 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-credential-socket\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.482737 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.482712 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:10:42.482858 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.482775 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "kube-api-access-r5hhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 08:10:42.482933 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.482868 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 08:10:42.482933 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.482866 5894 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy" pod "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" (UID: "54d37e1d-1a39-4f9c-8f4f-9163bd1197a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 08:10:42.582731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.582681 5894 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-podinfo\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.582731 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.582728 5894 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-envoy\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.583130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.582752 5894 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hhn\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-kube-api-access-r5hhn\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.583130 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.582771 5894 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3-istio-token\") on node \"ip-10-0-143-90.ec2.internal\" DevicePath \"\"" Oct 30 08:10:42.960028 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:42.959990 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-90.ec2.internal_1648b9f3a1682e8c87ccb70263aa6652/haproxy/0.log" Oct 30 08:10:43.039646 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.039590 5894 generic.go:356] "Generic (PLEG): container finished" podID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerID="59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15" exitCode=0 Oct 30 08:10:43.039866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.039665 5894 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" Oct 30 08:10:43.039866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.039670 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" event={"ID":"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3","Type":"ContainerDied","Data":"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15"} Oct 30 08:10:43.039866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.039735 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp" event={"ID":"54d37e1d-1a39-4f9c-8f4f-9163bd1197a3","Type":"ContainerDied","Data":"86ce2226bea4b4c00b683a7795603060d87cf9b9845ae46ed8d75b4e13eac4d3"} Oct 30 08:10:43.039866 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.039769 5894 scope.go:122] "RemoveContainer" containerID="59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15" Oct 30 08:10:43.051066 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.051044 5894 scope.go:122] "RemoveContainer" containerID="59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15" Oct 30 08:10:43.051376 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:10:43.051355 5894 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15\": container with ID starting with 59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15 not found: ID does not exist" containerID="59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15" Oct 30 08:10:43.051446 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.051393 5894 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15"} err="failed to get container status \"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15\": rpc error: code = NotFound desc = could not find container \"59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15\": container with ID starting with 59cb2d9eeb7bf5d17e01a504936dfb134062f0f37e058b3c3acb3dff4a807a15 not found: ID does not exist" Oct 30 08:10:43.113617 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.113571 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 08:10:43.128266 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.128227 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6dcf757899-767jp"] Oct 30 08:10:43.907812 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:43.907781 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:44.063161 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:44.063114 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:44.408908 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:44.408869 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:44.432041 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:44.432003 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" path="/var/lib/kubelet/pods/54d37e1d-1a39-4f9c-8f4f-9163bd1197a3/volumes" Oct 30 08:10:44.588415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:44.588376 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:45.570803 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:45.570761 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:45.615459 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:45.615421 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:46.256110 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:46.256076 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:46.300401 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:46.300363 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:46.363357 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:46.363321 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:46.408301 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:46.408266 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:47.012246 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.012201 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:47.029798 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.029764 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:47.059693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.059662 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:47.067498 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.067471 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:47.489917 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.489879 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:47.542944 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.542911 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:47.931375 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:47.931321 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:48.079510 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:48.079464 5894 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-4js78_5300a292-d959-4d7b-a1fe-b1a1f81efd71/konnectivity-agent/0.log" limit=1024 Oct 30 08:10:48.605012 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:48.604977 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:48.746007 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:48.745961 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:49.092853 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:49.092802 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:49.337547 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:49.337503 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:49.525919 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:49.525836 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5c2cx_01b69050-e4b3-483e-8065-0ef33a849871/manager/0.log" Oct 30 08:10:49.672832 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:49.672795 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-p6tlb_4864a195-5b08-4c9b-b91c-5312624dc72e/kuadrant-console-plugin/0.log" Oct 30 08:10:50.164683 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.164627 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:50.286468 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.286427 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:50.835293 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.835247 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wg9tq_13d4950f-8dc2-43d5-8969-842bc081873c/limitador/0.log" Oct 30 08:10:50.859260 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.859229 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:50.948187 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.948116 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:50.954721 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:50.954691 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:51.008600 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.008570 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:51.200332 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200151 5894 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv"] Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200775 5894 cpu_manager.go:414] "containerMap: removing container" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200801 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200844 5894 cpu_manager.go:414] "containerMap: removing container" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200853 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200875 5894 cpu_manager.go:414] "containerMap: removing container" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200884 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200900 5894 cpu_manager.go:414] "containerMap: removing container" podUID="a55febc7-7879-47b5-892a-766afc714734" containerName="revoke-keys" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200907 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55febc7-7879-47b5-892a-766afc714734" containerName="revoke-keys" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200923 5894 cpu_manager.go:414] "containerMap: removing container" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200932 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200956 5894 cpu_manager.go:414] "containerMap: removing container" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" Oct 30 08:10:51.200963 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.200963 5894 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201043 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c913759-de4e-40c6-8169-d3f511319963" containerName="payload-processing" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201054 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55febc7-7879-47b5-892a-766afc714734" containerName="revoke-keys" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201060 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d37e1d-1a39-4f9c-8f4f-9163bd1197a3" containerName="istio-proxy" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201068 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe76c9f-e2ef-471b-a29b-5d9caee9947a" containerName="main" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201076 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="973f8c38-94b3-4869-8df0-2691c6ecb509" containerName="main" Oct 30 08:10:51.201930 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.201084 5894 memory_manager.go:365] "RemoveStaleState removing state" podUID="c517bb18-ee3d-4b4c-a49d-911b92922de7" containerName="main" Oct 30 08:10:51.207150 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.207097 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.224230 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.224194 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv"] Oct 30 08:10:51.261900 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.261859 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-lib-modules\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.262136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.261919 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxgd\" (UniqueName: \"kubernetes.io/projected/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-kube-api-access-5kxgd\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.262136 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.261971 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-podres\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.262298 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.262185 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-sys\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.262298 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.262222 5894 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-proc\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.321528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.321412 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"openshift-service-ca.crt\"" Oct 30 08:10:51.321528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.321425 5894 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k6rrn\"/\"kube-root-ca.crt\"" Oct 30 08:10:51.321528 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.321455 5894 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k6rrn\"/\"default-dockercfg-lq6ls\"" Oct 30 08:10:51.362903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.362860 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-sys\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.362903 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.362910 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-proc\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.362957 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-lib-modules\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.362982 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxgd\" (UniqueName: \"kubernetes.io/projected/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-kube-api-access-5kxgd\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.362995 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-sys\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.363029 5894 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-podres\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.363050 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-proc\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.363125 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-podres\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.363219 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.363131 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-lib-modules\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.390536 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.390502 5894 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxgd\" (UniqueName: \"kubernetes.io/projected/9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb-kube-api-access-5kxgd\") pod \"perf-node-gather-daemonset-vlprv\" (UID: \"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb\") " pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.525071 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.524972 5894 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:51.632677 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.631578 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:51.695485 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.695443 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:51.718587 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.718548 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:51.777203 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.777116 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:51.847124 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:51.847083 5894 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv"] Oct 30 08:10:52.131183 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.131052 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" event={"ID":"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb","Type":"ContainerStarted","Data":"9e45e68c30892d45db242103e5ba210e1b4e85cb91fb9132c4ecd7ae0c1a1df2"} Oct 30 08:10:52.131183 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.131159 5894 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" event={"ID":"9eca3bd9-4df8-4169-b1ec-c01d0c2e3ebb","Type":"ContainerStarted","Data":"332e3819889b012a0ae79c487d9d50ff603fdc3dad9d70cb2e2931b67e7b2bc8"} Oct 30 08:10:52.131455 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.131239 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:52.178501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.178431 5894 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" podStartSLOduration=1.178392771 podStartE2EDuration="1.178392771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 08:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 08:10:52.177672205 +0000 UTC m=+4415.868589650" watchObservedRunningTime="2026-10-30 08:10:52.178392771 +0000 UTC m=+4415.869310209" Oct 30 08:10:52.259453 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.259395 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:52.313659 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.313597 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:52.323877 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.323848 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/alertmanager/0.log" Oct 30 08:10:52.440263 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.440170 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/config-reloader/0.log" Oct 30 08:10:52.558325 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.558287 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/kube-rbac-proxy-web/0.log" Oct 30 08:10:52.674571 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.674505 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/kube-rbac-proxy/0.log" Oct 30 08:10:52.790795 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.790762 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/kube-rbac-proxy-metric/0.log" Oct 30 08:10:52.907326 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:52.907299 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/prom-label-proxy/0.log" Oct 30 08:10:53.020693 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.020656 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:53.023694 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.023667 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6c2b4020-74f0-41d7-b93f-61f90f6296f1/init-config-reloader/0.log" Oct 30 08:10:53.162833 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.162743 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:53.287537 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.287478 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-988bg_3fc44517-a592-4c5e-9a42-96047cabe380/kube-state-metrics/0.log" Oct 30 08:10:53.393097 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.393068 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-988bg_3fc44517-a592-4c5e-9a42-96047cabe380/kube-rbac-proxy-main/0.log" Oct 30 08:10:53.512505 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.512427 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-988bg_3fc44517-a592-4c5e-9a42-96047cabe380/kube-rbac-proxy-self/0.log" Oct 30 08:10:53.648415 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.648381 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-65bd9d49b8-7bvx4_5a7f4421-c460-4690-b755-a6fef7f2e487/metrics-server/0.log" Oct 30 08:10:53.750702 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.750666 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jg24c_05999c41-8b41-470c-8762-87620709e24e/manager/0.log" Oct 30 08:10:53.775785 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.775699 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-65fv7_b1429333-ac51-49ee-8173-806b8ae7aeaf/monitoring-plugin/0.log" Oct 30 08:10:53.902923 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.902891 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82xqd_3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/node-exporter/0.log" Oct 30 08:10:53.939501 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:53.939455 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-hkjng_f0c68c94-cae9-476d-9c8d-560f6c998844/manager/0.log" Oct 30 08:10:54.019149 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:54.019118 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82xqd_3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/kube-rbac-proxy/0.log" Oct 30 08:10:54.136441 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:54.136413 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-82xqd_3ec28222-b8cd-4a03-8a22-3a0d81e5b50f/init-textfile/0.log" Oct 30 08:10:54.554465 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:54.554417 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:54.686204 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:54.686171 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:55.064878 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.064845 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lvcvs_dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/kube-rbac-proxy-main/0.log" Oct 30 08:10:55.186718 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.186683 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lvcvs_dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/kube-rbac-proxy-self/0.log" Oct 30 08:10:55.305324 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.305286 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-lvcvs_dbe05ba0-4fc7-4df3-bf3b-af4feadb519d/openshift-state-metrics/0.log" Oct 30 08:10:55.458117 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.458022 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/prometheus/0.log" Oct 30 08:10:55.477738 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.477699 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:55.533823 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.533788 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-2dkx9_85b51d8f-fd83-4c55-8b43-e830d2739434/manager/0.log" Oct 30 08:10:55.552025 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.551991 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:55.577893 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.577857 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/config-reloader/0.log" Oct 30 08:10:55.628857 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.628819 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/0.log" Oct 30 08:10:55.699283 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.699248 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/thanos-sidecar/0.log" Oct 30 08:10:55.821469 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.821432 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/kube-rbac-proxy-web/0.log" Oct 30 08:10:55.944201 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:55.944161 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/kube-rbac-proxy/0.log" Oct 30 08:10:56.063741 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.063696 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/kube-rbac-proxy-thanos/0.log" Oct 30 08:10:56.132134 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.132044 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:56.181216 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.181185 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddf96c98-f625-41ad-a303-63b2c4d92e95/init-config-reloader/0.log" Oct 30 08:10:56.190123 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.190074 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:56.320762 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.320718 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qd27n_799484a3-d828-4795-b681-710baad6a32c/prometheus-operator/0.log" Oct 30 08:10:56.339084 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.339038 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6454b597dd-mk5z8_896ba020-7327-4bc5-88ec-a2292da98092/manager/1.log" Oct 30 08:10:56.396497 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.396399 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-nlccq_6eb85c1d-f139-4bb3-a299-e9a7dae8a683/server/0.log" Oct 30 08:10:56.419513 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.419490 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-qd27n_799484a3-d828-4795-b681-710baad6a32c/kube-rbac-proxy/0.log" Oct 30 08:10:56.664316 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.664230 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8d7c96fcc-ltvtx_d076bc5b-c1f1-499c-8f92-9bb0b3e922b5/telemeter-client/0.log" Oct 30 08:10:56.781146 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.781060 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8d7c96fcc-ltvtx_d076bc5b-c1f1-499c-8f92-9bb0b3e922b5/reload/0.log" Oct 30 08:10:56.824483 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.824446 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-gqtn9_b4518ef6-492e-4c5b-8e5c-6bc7669a294d/manager/0.log" Oct 30 08:10:56.879913 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.879878 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-zmvtp_4c1f0382-bd87-4917-85bc-0054889e2c44/proxy/0.log" Oct 30 08:10:56.896949 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:56.896911 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8d7c96fcc-ltvtx_d076bc5b-c1f1-499c-8f92-9bb0b3e922b5/kube-rbac-proxy/0.log" Oct 30 08:10:57.020749 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.020625 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/thanos-query/0.log" Oct 30 08:10:57.135667 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.135618 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/kube-rbac-proxy-web/0.log" Oct 30 08:10:57.252013 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.251976 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/kube-rbac-proxy/0.log" Oct 30 08:10:57.368771 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.368746 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/prom-label-proxy/0.log" Oct 30 08:10:57.485395 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.485365 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/kube-rbac-proxy-rules/0.log" Oct 30 08:10:57.602829 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:57.602799 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-58f5c47444-cfzwr_99c07bc5-c5be-4d09-9f96-1e50f37423be/kube-rbac-proxy-metrics/0.log" Oct 30 08:10:58.152220 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:58.152187 5894 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k6rrn/perf-node-gather-daemonset-vlprv" Oct 30 08:10:58.425542 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:58.425446 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nf7b5_e60bb623-53ca-42c5-af7d-610ee177be92/discovery/0.log" Oct 30 08:10:58.461463 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:58.461415 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-5mvd9_35207e99-083b-4214-bf26-79fabec5c2bf/kube-auth-proxy/0.log" Oct 30 08:10:58.736678 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:58.736563 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vpdnn_cd6421ad-3f96-4128-9aae-9bd4ba795a75/istio-proxy/0.log" Oct 30 08:10:59.041250 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.041216 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v"] Oct 30 08:10:59.055720 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.055681 5894 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl"] Oct 30 08:10:59.060504 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.060273 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c66bf60-v288v"] Oct 30 08:10:59.063583 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.063555 5894 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e67fd566-t46nl"] Oct 30 08:10:59.207274 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.207235 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nf7b5_e60bb623-53ca-42c5-af7d-610ee177be92/discovery/0.log" Oct 30 08:10:59.240020 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.239983 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-5mvd9_35207e99-083b-4214-bf26-79fabec5c2bf/kube-auth-proxy/0.log" Oct 30 08:10:59.508484 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:10:59.508398 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vpdnn_cd6421ad-3f96-4128-9aae-9bd4ba795a75/istio-proxy/0.log" Oct 30 08:11:00.430086 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:00.430046 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3050a307-8385-4ecf-b84a-20f6396d8afb" path="/var/lib/kubelet/pods/3050a307-8385-4ecf-b84a-20f6396d8afb/volumes" Oct 30 08:11:00.430712 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:00.430691 5894 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b9a27a-647a-4098-94ca-7372355d9282" path="/var/lib/kubelet/pods/53b9a27a-647a-4098-94ca-7372355d9282/volumes" Oct 30 08:11:04.064410 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:04.064336 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5c2cx_01b69050-e4b3-483e-8065-0ef33a849871/manager/0.log" Oct 30 08:11:04.123766 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:04.123739 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-p6tlb_4864a195-5b08-4c9b-b91c-5312624dc72e/kuadrant-console-plugin/0.log" Oct 30 08:11:04.325934 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:04.325837 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wg9tq_13d4950f-8dc2-43d5-8969-842bc081873c/limitador/0.log" Oct 30 08:11:04.832909 ip-10-0-143-90 kubenswrapper[5894]: E1030 08:11:04.832862 5894 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mwst9" containerName="extractor" Oct 30 08:11:04.966698 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:04.966666 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-5c2cx_01b69050-e4b3-483e-8065-0ef33a849871/manager/0.log" Oct 30 08:11:05.030621 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.030585 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-p6tlb_4864a195-5b08-4c9b-b91c-5312624dc72e/kuadrant-console-plugin/0.log" Oct 30 08:11:05.079918 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.079880 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55nz9_6215b7c5-298f-481a-b3ce-884ad7382b26/dns/0.log" Oct 30 08:11:05.201061 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.200978 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-55nz9_6215b7c5-298f-481a-b3ce-884ad7382b26/kube-rbac-proxy/0.log" Oct 30 08:11:05.209839 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.209807 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-wg9tq_13d4950f-8dc2-43d5-8969-842bc081873c/limitador/0.log" Oct 30 08:11:05.498760 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.498671 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nf7b5_e60bb623-53ca-42c5-af7d-610ee177be92/discovery/0.log" Oct 30 08:11:05.531050 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.531021 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cf9d6756f-5mvd9_35207e99-083b-4214-bf26-79fabec5c2bf/kube-auth-proxy/0.log" Oct 30 08:11:05.787348 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.787312 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6dcf757899-vpdnn_cd6421ad-3f96-4128-9aae-9bd4ba795a75/istio-proxy/0.log" Oct 30 08:11:05.963201 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:05.963170 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wdkkw_9e4f6bc7-4192-4165-b16d-df0db49aedd9/dns-node-resolver/0.log" Oct 30 08:11:07.612929 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:07.612895 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhhvm_20e60768-4e85-440c-8ddf-cb8f91fdd289/node-ca/0.log" Oct 30 08:11:10.250259 ip-10-0-143-90 kubenswrapper[5894]: I1030 08:11:10.250221 5894 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nf7b5_e60bb623-53ca-42c5-af7d-610ee177be92/discovery/0.log"